Down and Dirty

When modeling problems for more useful results, the harder they come, the harder they fall

Occam's razor says the simplest answer is probably right. this works in physics. But when it comes to superior modeling of complex, human-created domains, although at some high level the method may look elegant, at a detailed level the modeling is going to require some down-and-dirty complexity.

Down and dirty may be the best description for the next generation of hard problems we face, which the next generation of decision-support systems will need to solve. Let's think about what these problems are, why they are hard, and why solving them might not be as elegant as we want.

When you dig for deep understanding of human constructions - languages, economies, unique personal events and demands, business deals, complex financial and insurance contracts, and so on - you get more from your investigation if you have an eye for uniqueness and context. For example, the value of a term in a given circumstance is bound by its history of use among you and humanity, its nearby words, the sentences above and below, and the documents you place it in.

The same attributes that would make decision-support solutions tremendously valuable - the ability to understand and integrate many different levels of detail - also make them tremendously difficult to build. And if you don't face the fact that the solution methods have to be complex, you will just get stuck on some cloud nine theory with results of limited use.

As an example of simple solutions leading to limited use: Most of what we call "personalization" today really is just a bit of market segmentation; you still get statistics, just from a smaller, yet still large group. This kind of personalization is very different from the kind you get when walking into a tailor's shop you have been patronizing for years where the employees know you well. "Good to see you, Mr. Jones. It looks like you've lost weight. Have you been working out? I have the perfect suit for you."

It's great to receive special treatment. Many different population groupings might seem to typify Mr. Jones by themselves, but in total they might offer a much higher level of personalization. Mr. Jones, for example, might be a dieter, a member of a health club, a person who cares about his clothes, and someone who enjoys being flattered. Putting all these parts together offers the greatest insight on how you might best interact with this individual - perhaps the next best thing to a long-term face-to-face relationship.

Just as you can argue that all politics are local, we often want business decisions to have the option of being local and specific, while simultaneously taking account of how that fits into different or larger views. The ultimate decisions should be allowed to reflect tracking, predicting, as well as taking advantage of local or specific circumstances and larger circumstances at many different levels or kinds of aggregation all at the same time. For example, we might want to consider trends in taxation, competition, product availability, and preferences, all at the local, regional, or countywide levels. Likewise, predictions at various time frames, which can look very different, should be able to be combined.

The statistician's first instinct - to get better estimates on bigger samples - runs counter to trying to evaluate the probable and multiple desires of small groups, individuals, or locations. This problem is analogous to many others, including enabling a computer to see. Vision by its very nature starts at the local level. Each receptor (such as a retinal cell) receives different information - light or dark elements of the image. Then out of these fragmented facts, the mind extracts larger properties, such as the existence of edges, until it organizes or fills in enough things through extension and generalization that it creates and recognizes the relationships between high-level and localized reconstructions of the world. Thus with a brief look we can instantly recognize a whole face, its parts - the eyes, nose, mouth, and so forth - and how these parts fit in to make the whole.

The need for specialized, localized analyses to happen simultaneously with integrative analyses also applies when looking for improvements in larger knowledge management solutions. Word values or perhaps even meaning - whether derived from strict definitions of terms, situational nuances, grammatical roles, or the implications of formats - is always highly dependent on context. Much information is implicit or, worse, not immediately available at all.

Because decision-support solutions always entail extrapolating from what little may be known, we might then consider the basic standard of "intelligence" to be the ability to see correctly one step beyond what is given. But making these extensions requires setting assumptions, whether implicitly or explicitly. The more appropriate these assumptions are, the more related the system is to the real world and therefore the more useful the system is.

However, more often than not these assumptions need to be derived from, made in tandem with, or confirmed with analyses at additional levels. So overall intelligence clearly depends on this relatedness working simultaneously and somewhat consistently at different levels.