Down and Dirty

When modeling problems for more useful results, the harder they come, the harder they fall

Thus the intelligence of any such system depends not only on the relatedness of each component to the real world; it also relies on how the components stack up to model relationships in the problem domain - ideally reinforcing correct assumptions so the parts sum up to a greater intelligence.

No single uniform method is going to solve these kinds of problems. Solving them will require methods that match the variability of component problem demands as well as the demand for their integrated analysis and organization - themes I have been covering in this column over this past year. Rather than seeing business intelligence (BI) methods converge to become mega-solution systems, we will continue to find in 2001 great room for the innovators, startups, and creative vertical solution integrators as they begin to build or combine what is needed for various problem segments.

However, we can get some insight, even an overall language, that helps us think about many of these problems, by looking at how nature has solved similar sorts of complex information-processing problems. The solutions found in ant colonies, brains (such as sense and motor capabilities), DNA, and the human language ability have something in common. They take the form of many specialized units (or agents) solving specific problems limited in space, time, or complexity, with each requiring localized assumptions. These agents in turn are organized by higher-level units (also agents), working up to the overall system properties needed for success.

It seems to me this solution architecture is just a consequence of the nature of these kinds of complex problems - multilevel problems require multilevel solution methods. So I expect more and more solutions in BI to take the form of large intelligent systems of intelligent systems (agents) with specialized components for each subproblem. Developers or users will have tremendous flexibility in organizing these components to build to their needs. The demands of e-commerce, speech recognition, and Web-based answer discovery will be driving forces.

Using my columns from last year, I tried to lay a foundation for what it might take to solve some of these problems, which are actually amalgams of many different component subproblems. (And organizing the subproblems offers yet another set of problems). In 2001, I expect we will see ever more successful applications with related techniques, with even more pioneering solution methods emerging.

These methods include ever more powerful ways of creating specialized systems of Markov models with customized, situation-related transitions and special architectures of agents for mining complex, unstructured sources. Those sources include text, audio, pictures, and video. New function forecasting and categorization techniques, as well as agents with knowledge of specialized ontologies, will be components of these solutions.

Additionally, I expect to see more and more automated techniques for both producing and combining the required agents.

I believe there will be greater recognition that some of the techniques already completed or in development to classify, forecast, and function fit will also work to automatically produce and combine agents. We will also gain greater understanding of how to automatically segment problems so as to apply these solution approaches iteratively.

Plus we may see more work with recombinant processes - those that in some ways mimic DNA - in producing smart units and their rules of interaction, and testing their variations in order to automatically construct the complex systems we require. Intelligent drag-and-drop GUIs that suggest organizations of components may be only the beginning.

Just because each solution is specialized does not mean it has to be created separately. The next generation beyond CASE program development tools will undoubtedly include automated recombination, variation, and testing methods, perhaps even intelligent ones, that attempt to have a computer automatically build software systems to fit pre-set functional specifications.

We misunderstood the complexity of artificial intelligence problems - we thought they were easy and quickly solvable. At least we are beginning to get a more realistic grasp of what it will take to create solutions. Once again we will see more and more new methods from AI driving solutions in BI.