Software Commodification

Though there's been a lot of talk about software commodification (and commoditization), I've never seen the issues formulated simply, so here goes...

A commodity is any product which meets some agreed upon criteria, such that any instance of that commodity may be substituted for any other. A quart of milk from any supplier will satisfy your cupcake recipe, no matter who produces it, so milk is a commodity. A necktie, however, is subject to all sorts of weird style criteria, so it is not a commodity. It might not even be your color.

All software programs are commodities in essence. Programs follow logical rules and perform to a specification, but we can't say a given program is a commodity before it has an actual substitute. So commodity nature is revealed when a given piece of software acquires competing alternatives.

Most software alternatives are not perfect substitutes in the sense that you can switch implementations and barely notice. Usually switching requires retraining the people that use the product and porting the software that uses the product. Vendors obviously want to keep switching as difficult as possible, and techniques for doing so are called "lock-in". By definition, strong lock-in hurts the consumer, but it also usually hurts the consumer by creating an inferior product. A product with strong lock-in will generally be monothilic, have complicated interfaces, and have hidden dependencies.

Given a choice, anyone (whether selling software or anything else) would prefer to have their product not to be a commodity. This is because markets are especially efficient at setting commodity prices, and this means suppliers will have stiff competition.

It's worse to have your software product become a commodity. This is because of the nature of software (and any non-physical product). The first copy of your software will cost you $2,000,000 but every subsequent copy will cost you only $2. This means that when someone enters your market, they've already done the hard part, and there's no incentive to leave. More importantly, the price of a software product is completely arbitrary. A price war could bring prices all the way down to zero.

Free competitors may be particularly formidable. Only with software is free competition realistic, because with physical products, revenue must reliably exceed costs. Free software makes it particularly easy to pool contributions from disparate sources, with very little administrative overhead, and very little reliance on a single controlling entity. (If a maintainer isn't doing a sufficiently good job, then someone may "fork" the development effort.) Free efforts also generally aren't motivated to lock-in.

All things being equal, it's desirable for software vendors to have the other software that they depend on to be commodified. This is for a few reasons: your dependencies have power over your product, and when your dependencies are cheaper, your overall product solution will be cheaper. There'll be a larger market for your product.

So if free software is so easy and desirable, how can a commercial software vendor stay in business? By presenting a moving target. Any given software release may be straightforwardly commodified, but the release process may not be. No entity or group of entities will have external incentive to track a series of upgrades. This is partially why "tying" is so important to a large software vendor, because tying is the only way for a mature product to escape commodification.

Ironically, the vendor is really in the business of selling timely upgrades, a service business which uses the base software product as merely a barrier to entry.

Intuitive Interfaces

An interface can be intuitive for a many reasons:

• It works like other similar interfaces that the user has seen. (Green means "go".)
• It's logical. (Cars' signalling interface correspond to the way that you turn the wheel.)
• It's internally consistent. (The same abstraction works the same way in different contexts.)
• It has a shallow learning curve.
  • It has helpful pointers, ideally inobtrusive. (Examples are popup help bubbles and pie menus)
  • It allows safe experimentation. (You can "undo" any operation.)
• It's simple. :)

Python Evolution and AOP

An interesting trend in the evolution of the python language is that it has acquired ways of hooking into more and more code points that are not traditionally hooked into.

For a while, python has had metaclasses, which allow customizing the behavior of whole classes of classes :). These metaclasses allow running code on class creation, and routinely involve customizing object creation.

In order to obviate java's setter/getter pattern, python has the ability to run code on getting an attribute or setting an attribute or both.

Recently, python attained a decorator feature, which allows running code on function/method creation. This feature was originally motivated by wanting to specify things like a method being "static".

Now there's talk about implementing type-checking use adaption of function/method arguments and return values.

This trend could seem a lot like Aspect Oriented Programming, but it has the important difference that the path to the hook code is always evident at the hooked code. A main point of AOP is perfect "obliviousness", according to which you can alter the behavior of foreign code without making any modification to it. In python, the main code body can be oblivious of the hook, but it's always preceded by an explicit reference to the hook, almost like an annotation.

In python, each of these hooking mechanisms has a different syntax, presumably tailored to its primary use case. These features are mostly made possible by python's easy introspection support, and by its uniform "object" treatment of all of these abstractions.

Managing Change

It's time to release an upgrade to your whiz-bang widget library. If you want to avoid breaking all the code that people have written to use and even extend your library, here's what to do.

Obviously, don't remove public and protected methods, or change their semantics. For protected methods, this includes having them continue to call other nonprivate methods under the same conditions.

Less obviously, if you're creating new public or protected methods, you should move your code to an entirely new class! Someone may have extended your class with a method that has the same signature that you've just added, causing a needless conflict. Under the old class, create proxy classes that delegate every method to the new code. You don't have to worry about the maintenance overhead, because these proxies represent the legacy interface; you shouldn't have to change them later.

This hints at the fact that your library should be very careful about calling its own nonprivate methods. Calling internal nonprivate methods should be reserved for supported abstractions, because your library will down-call into extensions of these methods. If you don't want to support the abstraction, move the implementation into a private method. Have your internal code call the private method, and have your public method just be a wrapper for clients.

This approach is not so elegant because it requires code changes in order to support unchanging function. Maybe languages should support using interfaces for encapsulation, so that any symbols not present in the specified interface(s) would be completely hidden. However, all the recent python typechecking and adaption thinking seems to be focused on method calls, not base class specification.

Minimum Curriculum

Everyone should learn the most important ideas of human civilization. There should be a year-long lecture course that introduces all the "must see" ideas, the best that the human race has found, the stuff that would be tragic for a person to have lived and never learn. Certainly many things require lengthy study to be useful, and many things are already as clear as day to many people, but there are many important ideas of which educated people are ignorant.

Even if this makes sense for ideas, it certainly makes no sense for art. The survey courses in dance, art history, music history, and world literature probably can't be meaningfully condensed any further. But no one should miss out on the major ideas of the introductory courses in math, astronomy, physics, chemistry, biology, psychology, economics, computer science, philosophy, political science, philosophy, and history. Some ideas fall through the cracks of all of these intro courses. (For example, what about the central points of The Death and Life of Great American Cities?) Some ideas are also too new to make it into conventional courses. For many ideas, only a few minutes explanation would be sufficient.

Many survey courses take the historical approach, but in this context it would waste too much time. To make it more interesting, the course (and more importantly the reading list) could touch on the limits of human knowledge and current work.

I originally had this idea in yeshiva, in a spirit of parochialism: "take this course, and then forget about the rest." Ironically, the course would probably not be popular with the insular (even if it were to omit evolution). This idea came back to me recently, thinking about John Rawls. I realized that neither my college survey in humanities nor intro to philosophy mentioned him.

The following are the ideas that I could come with quickly; please suggest more.

big bang, scale and elements of the universe, basic probability, limits, idea of calculus, newton's laws, four forces, electricity, nature of light, relativity, phases of matter, entropy, periodic table, cells, virii, bacteria, dna, systems of the body, evolution, milgram study, the unconscious, supply and demand, how a computer works and what a program looks like, urls, kant's categorical imperative, veil of ignorance, a myriad of -isms: utilitarianism, facism/socialism/capitalism(democracy), world religions, epistemology, problem of induction, logic and godel for dummies, mixed-use cities, human history in 30 minutes :), how toilets work

G-d Exists, so I could be Wrong

From yesterday's new york times article "God (or Not), Physics and, of Course, Love: Scientists Take a Leap":

David Myers
Psychologist, Hope College; author, "Intuition"

As a Christian monotheist, I start with two unproven axioms:

1. There is a God.

2. It's not me (and it's also not you).

Together, these axioms imply my surest conviction: that some of my beliefs (and yours) contain error. We are, from dust to dust, finite and fallible. We have dignity but not deity.

And that is why I further believe that we should

a) hold all our unproven beliefs with a certain tentativeness (except for this one!),

b) assess others' ideas with open-minded skepticism, and

c) freely pursue truth aided by observation and experiment.

This mix of faith-based humility and skepticism helped fuel the beginnings of modern science, and it has informed my own research and science writing. The whole truth cannot be found merely by searching our own minds, for there is not enough there. So we also put our ideas to the test. If they survive, so much the better for them; if not, so much the worse.

Core Assumptions

A college friend gave me a compliment; he said that when I argue with someone, I persist only until discovering his assumption that conflicts with my assumptions.

Wouldn't it be great if all disagreements could be reduced to conflicting core assumptions? If the parties could agree on this reduction, then maybe the assumptions themselves could be meaningfully discussed and compared.

Of course, I'm thinking of the electoral split of recent times. Could it be that conservatives simply don't accept the idea of the Veil of Ignorance?

Beating Java on Its Own Turf

The author of python, Guido van Rossum is writing some great stuff about type-checking in python, which will enable even more specific contracts.

This is exciting because it promises to achieve the safety of java with the simplicity of python. Safety is one of the three technical reasons that people still use java instead of python. The other two are performance and enterprise software. Performance is being addressed by the newly funded PyPy. Python already has much of the java enterprise technology, just not in the form of ballyhooed specifications. (For the remaining technologies, it's not clear that the "enterprise solution" achieves better reliability or scalibility than folks' nonstandard solutions.)

That the business community seems to care more about these specifications than about the actual technology may be for the same reason that it doesn't greatly prefer Sun's specifying Java over Microsoft's owning .NET. (Without a patent grant, the incomplete .NET standards work is almost worthless.) One would think that having one company write specifications for which many companies can build implementations would be much safer, and would create much more open software. That the wholely proprietary .NET is even considered a contender implies that people only care about the company that writes the interfaces.

Special Characters and Gmail

Not only does gmail filter out ".", it also chops off everything from the first "+".

This enables using tmda or simply filtering on what address you choose to give. (Similar to how some people give vendors their names with different middle initials, in order to track how their name got onto different lists.)

JOtherComboBox

Java is a little wack. Its selection box gui widgets have items which are either editable or not. The following is the necessarily verbose way to create an uneditable selection box that allows adding a new item with an explicit action. (Contrast with autocompletion in an editable field, which provides a very low barrier to adding new items.)

import java.awt.Component;

import java.awt.event.ActionEvent;

import java.awt.event.ActionListener;

import java.awt.event.KeyEvent;

import java.awt.event.KeyListener;

import javax.swing.JComboBox;

import javax.swing.JFrame;

import javax.swing.ComboBoxModel;

import java.util.Vector;



/**

*

* JComboBox for which selecting a particular item allows editing a new item

*/

public class JOtherComboBox extends JComboBox {

	int otherIndex;

	ActionListener otherItemActionListener = new ActionListener() {

		public void actionPerformed(ActionEvent e) {

			JOtherComboBox widget = JOtherComboBox.this;

			if (widget.getSelectedIndex() == otherIndex) {

				widget.setSelectedItem(null);

				widget.setEditable(true);



				widget.requestFocus();

				widget.getEditor().getEditorComponent().addKeyListener(new KeyListener() {

					public void keyTyped(KeyEvent e) { work(e); }

					public void keyPressed(KeyEvent e) { work(e); }	

					public void keyReleased(KeyEvent e) { work(e); }

					public void work(KeyEvent e) {

						((Component)e.getSource()).removeKeyListener(this);

						e.consume();

					}

				});						

			} else if (widget.getSelectedIndex() >= 0) {

				widget.setEditable(false);

			}

		}

	};

	

	public JOtherComboBox(Object[] arg0) { super(arg0); init(); }

	public JOtherComboBox(Vector arg0) { super(arg0); init(); }

	public JOtherComboBox(ComboBoxModel arg0) { super(arg0); init(); }

	public JOtherComboBox() { super(); init(); }



	public void init() {

		this.addActionListener(otherItemActionListener);

	}



	public int getOtherIndex() {

		return otherIndex;

	}

	

	public void setOtherIndex(int arg0) {

		otherIndex = arg0;

	}



	public static void main(String[] args) {

		JFrame frame = new JFrame();

		JOtherComboBox comboBox = new JOtherComboBox(new String[] {"one","two","three","four"});

		comboBox.setOtherIndex(2);

		frame.getContentPane().add(comboBox);

		frame.pack();

		frame.setVisible(true);

	}



}

Databases Suck

That is, what we refer to as relational database management systems suck. They're too monothilic. An RDBMS typically has

• a persistent store
• a type system
• indexing features
• a transaction manager
  
• a query language
• a network protocol
• usually a full embedded programming language
  

All this stuff makes them inflexible, and painful to integrate. People like them because they're completely independent of the rest of your code, even to the extent that a human being can often poke at them without application-specific software. They're also standardized-ish.

One Device

Even if single-purpose devices often work better, when it comes to mobile devices, people want one device that does everything. Otherwise, it can get to be a lot to carry around.

So here are the functions that are candidates for integration:

• telephone
• camera
• image browser
  
• audio player
• storage media
  
• broadcast radio receiver
• audio recorder
• web browser
• email client
  
• organizer
• video player
• video camera
  

Most of these have already been integrated into at least one cell phone product. When will we see it all come together? What a device to have hackable!

These super-devices will change the world. In order to do it, they'll...

• use good voice recognition as the primary user interface
• embrace a more end-to-end approach

Cell phone technology is still very far from end-to-end. Carriers will try to prevent the change, because it will completely overturn their businesses.

Note about Gnome Storage

Not only will Gnome Storage make finding documents easy, remove the need for "Save", and therefore simplify task management, it will also replace a lot of existing interfaces.

Bookmarks, browser history, email indices, song playlists, photo galleries, "personal information management" views: working with these interfaces composes a large part of the time we spend with computers. Once all their info goes in a common storage layer, there'll be no reason to have many different metadata viewer implementations. (Of course, there'll be a mechanism to customize views.)

But that's not all. The desktop'll get many of the advantages of the web. We'll be able to create links to any object, whether it's an email message, a song, or a todo item. Any object may be bookmarked, and we'll get a history of all the objects that we've viewed.

This means big things for the semantic web movement, because it'll create huge amounts of metadata. Not only will we suddenly have easy, standard access to how often we listen to our favorite songs, we'll also know that they're predominantly in the "mp3s from joe" query folder, for example.

Once most applications can store their data in common, we can give them a common implementation of other fun features, versioning and notification, collaboration and localization.

No Password for Local Users

Do you want to relieve local linux users of having to enter passwords? Follow these steps to disable default remote access, which will make it safe for default accounts to have no password.

1. create a group "remote", and add users to it that may login remotely
   
2. add the following lines to your /etc/security/access.conf:

   +:remote:ALL
   -:ALL:localhost
   +:ALL:LOCAL
   -:ALL:ALL
   

3. add the following line to your /etc/pam.d/system-auth:
   

   account     required      /lib/security/$ISA/pam_access.so

4. add the following line in your /etc/pam.d/su:

   auth       required       /lib/security/$ISA/pam_wheel.so use_uid deny group=remote

If you have a local domain, you might have to tinker more with access.conf, because the "LOCAL" source specification is funny; it matches any host or tty name without a ".".

This was tested with Fedora Core 2.

Of course, you could take the converse approach: give everyone a password, and just don't require passwords for local logins to accounts in the "local" group. This approach is probably more robust against configuration changes (e.g. if a service stops including "system-auth" it will allow logging in without a password). The first approach however is nice for granting minimum necessary privilege. Maybe both is best. Presumably the second approach would be done with a line in local service pam.d config:

auth       sufficient   pam_listfile.so item=user sense=allow file=/etc/localusers onerr=fail

and this works for gdm at least. "mingetty" (for text logins) is a problem because it shares the "login" pam config with "in.telnetd" for plain-text remote logins. Of course, you could hack it with pam_access again (and /etc/security/group.conf) , but the extra complexity probably isn't worth it.

My Daughter

My daughter is the coolest. She just turned three. A couple of days ago she was watching the Elmo episode about music. ("Elmo" is a friendly, red furry creature, on a segment of Sesame Street for toddlers. Each Elmo episode has a theme.) Elmo's pet goldfish, Dorothy, had a toy drum in her jar. My daughter observed, "Dorothy can't play the drum because she don't got hands. She also don't got a stick."

Gcc on Solaris

If you're getting

/usr/include/sys/termios.h:376: parse error before `uint32_t'

then you should be invoking gcc as:

gcc -I/usr/include

Better Spam Protection

In short, it's a combination "disposable addresses" and "challenge/response whitelists"...

1. Everyone in your addressbook can mail you without qualification
2. Everyone can respond to one of your private messages
3. Everyone can respond to one of your public messages within a fixed interval of time
   
4. Everyone else can mail you by confirming their first message by email

The "whitelist" refers to #1, and it may also incorporate the address of anyone new that you email. It's insufficient because many people send mail from a different address than they receive mail. #2 solves this problem by embedding a cookie in your return address (e.g. JoeSmith+keyword+8769d5.4f88c3@someISP.com). (Many ISPs allow you to receive messages to your account name appended with a plus or minus followed by any string.)

You can embed a hash of the address that you're sending to, plus a hash of that hash. The second hash allows you to check that the embedded cookie is valid. The first hash allows you to create a single return address for each address that you send to. If someone who knows that address ever sends you spam, you can blacklist that generated address only.

When you want to receive responses to a publically posted address, it isn't a question of whether you'll want to eventually blacklist the address, but when. In that case #3, you can embed a cookie which instead contains the date until which you'll accept unconfirmed messages.

Even people who can't use the above methods should be able to reach you #4. All of the those people must undergo the slight annoyance of confirming their first message. Their first message will trigger an automatic response from you explaining the situation. They must respond once to that message, and they'll then be added to your whitelist.

The TMDA program does almost all of the work. #2 requires a two line patch. TMDA out-of-the-box is completely unconfigured, but my configuration looks pretty portable. I'll post it after I've been using it for a bit.

Future Interfaces

Computer interfaces should be dramatically improved, in the spirit of mpt's crufty rant. He sounds a refrain "We have the technology, so why....?" I don't think we have the technology yet, but we should.

No one should ever have to "save" a file. This requires a few things:

1. Unlimited undo and redo
2. Files should only be named with UUIDs
3. Files should be easily searchable using metadata

Once we've eliminated the save requirement, we can eliminate the unnatural concept of "open" files and windows. The only natural concept is whether a file is visible or not. Everything else should be a transparent optimization.

We can't just use unix inodes instead of UUIDs for two reasons: they don't work across multiple filesystems and they can't handle being removed and restored. I don't think there is a stable implementation of the metadata interface yet, though Gnome Storage looks promising.

Finally, applications will need to support transparent persistence. We can't just use generic checkpointing because it would be so wasteful. (For example, a web browser keeps a lot of stuff in memory, but it really only needs to persist a scrollbar location, a url (and maybe a cached copy of the url, which it'll need apart from the checkpointed image anyway).) Window managers would also have to be better at managing and persisting window arrangements.

Besides window sizes and locations, we'd like to manage tilesets; we'd like them to remember the fact that we're working with two documents side-by-side. Tzorech iyun, sometimes tilesets just indicate the desire to work on two different things at the same time.

Traits for Python

Since python already has multiple inheritance, implementing traits isn't a matter of adding new functionality, but of restricting the functionality that's already there.

The metaclasses below enable preventing classes from having member data and preventing conflicting methods. (Just have your classes inherit from Trait or TraitUser.)

The next step is solving the Artistic Cowboy Problem in a pythonic fashion.

If you missed my earlier traits entry, here's the traits homepage link again.

#!/usr/bin/python

import operator, types, sets



def get_classdict(cls):

        "get name:attr dict from a class"

        names = dir(cls)

        classdict = {}

        for name in names:

                classdict[name]=getattr(cls, name)

        return classdict



def get_methods(classdict):

        "get a set of nonsystem methods from a classdict"

        methods = sets.Set()

        for (k,v) in classdict.items():

                if k.startswith('__'): continue

                if type(v) is types.MethodType or type(v) is types.FunctionType:

                        methods.add(k)

        return methods



class TraitClass(type):

        """Metaclass for classes that may not have state,

        whose conflicting trait methods must be resolved in extenders,

        and who may not extend classes which are not traits."""



        def __init__(cls, clsname, bases, classdict):

                type.__init__(cls, clsname, bases, classdict)

                cls.conflict_methods = sets.Set()

                all_methods = sets.Set()

                for c in cls.get_method_lists(bases, classdict):

                        methods = get_methods(c)

                        cls.conflict_methods |= all_methods & methods

                        all_methods |= methods

                #propagate conflicting methods from base classes

                for c in bases:

                        if 'conflict_methods' in dir(c):

                                cls.conflict_methods |= c.conflict_methods

                for c in bases:

                        #if c == object: continue

                        if type(c) != TraitClass:

                                cls.handle_base_class(c)



        def get_method_lists(cls, bases, classdict):

                "traits check for conflicts from parents and from self"

                return map(get_classdict, bases)+[classdict,]



        def handle_base_class(cls, c):

                "enforce only inheritting traits"

                raise "Trait %s cannot inherit from a class that is not a Trait"%cls.__name__,c.__name__



        def __new__(cls, clsname, bases, classdict):

                #if a nonexplicit class extends nontraits, make it a trait user

                if classdict.get('__metaclass__') != TraitClass and 0 < len([c for c in bases if type(c) != TraitClass]):

                        classdict['__metaclass__']=TraitUserClass

                #if class is actually a trait, make readonly

                else:  

                        classdict['__slots__']=[]

                return type.__new__(cls, clsname, bases, classdict)



class TraitUserClass(TraitClass):

        """Metaclass for mostly normal classes,

        that must use single inheritance of nontraits,

        and that implement trait conflict handling"""



        def __init__(cls, clsname, bases, classdict):

                TraitClass.__init__(cls, clsname, bases, classdict)



                unresolved = getattr(cls, 'conflict_methods',sets.Set()) - sets.Set(classdict) - sets.Set(dir(getattr(cls,'nontrait_base',None)))

                if len(unresolved) > 0:

                        raise "conflicting methods",unresolved



        def get_method_lists(cls, bases, classdict):

                "check for conflicts in parent traits"

                nontrait_base = getattr(cls, 'nontrait_base', None)

                return [ get_classdict(c) for c in bases if c != nontrait_base]



        def handle_base_class(cls, c):

                "enforce single inheritance of nontraits"

                nontrait_base=getattr(cls,'nontrait_base',None)

                if nontrait_base:

                        raise "Trait-using class %s can extend no more than one class that is not a Trait"%cls.__name__, c.__name__

                cls.nontrait_base = c



        def __new__(cls, clsname, bases, classdict):

                #python weirdness; __metaclass__ doesn't override parent's here

                if classdict['__metaclass__'] == TraitClass:

                        raise "Trait %s cannot inherit from a class that is a TraitUser"%clsname 

                return type.__new__(cls, clsname, bases, classdict)



class Trait: __metaclass__=TraitClass #convenience object

class TraitUser: __metaclass__=TraitUserClass #convenience object

Python Metaclasses are Weird

$ python <<''

> class m1(type): pass

> class m2(m1): pass

> class c2: __metaclass__=m2

> class c1(c2): __metaclass__=m1

> print 'the type of c1 is', type(c1)

>

the type of c1 is <class '__main__.m2'>