Intelligent Enterprise

Better Insight for Business Decisions

RSS
Webcasts
Whitepapers
Subscribe
Home

November 12, 2001

The Quest for Speed

More users, more data, and more complex analytic demands are requiring new kinds of software

Barry Grushkin

The wild-looking mathematics professor ran into the seminar and proclaimed enthusiastically, "I have an invention that will revolutionize mathematics!" On two separate boards, he proved two separate theorems - one with each hand - each at his usual hypercharged speed.

Although what I saw that fall morning at Cornell University has not since become widely popular, it foreshadowed a trend in business intelligence (BI) systems: The demand for ever faster and more complex analyses on ever larger data sets is requiring parallel processing and special architecture tricks to crank up response speeds.

CONVERGENT EVOLUTION?

The human brain (not just the wild math professor's) is dominated by parallel interacting circuitry. To overcome the slow processing speeds of nerve cells, brains evolved problem-specific circuitries that can run simultaneously to solve ever more complex challenges. The 300-million years between trilobites and humans is a lot of time to experiment with new modes of sensory analysis.

Computer science, on the other hand, started with a different approach. The idea that even sophisticated processes could be broken down into a small set of simple sequential steps was entrenched in Western mathematics and philosophy long before being formalized into the theoretic notion of a Turing machine, which in turn existed before any relay or vacuum tube.

THE FASTEST SUPERCOMPUTERS

BUSINESSES ARE INCREASINGLY USING SOME OF THE WORLD'S FASTEST SUPERCOMPUTERS

The Web site Top500.org ranks the world's fastest supercomputers. The vast majority has been constructed with research questions in mind, but national defense reliability testing, weather forecasting, code breaking, and geological exploration are clear examples of applications with high computation demands. As for business applications, interestingly, Charles Schwab & Co. is 20th on the list, with a supercomputer most likely used to price complex derivatives with time-series Monte Carlo techniques and for trading strategy research. Some of the other businesses using the world's fastest computers, besides the expected oil companies, are American Airlines Inc. (122nd), Deutsche Telekom AG (128th), and Philip Morris Companies Inc. (138th).

Although hardware computing speeds took off exponentially from this point, the complexity of problems increased even faster. Particularly in science and technology research, parallel processing - the idea of organizing many serial processors to work in tandem - has become an essential component of solution architecture. For many complex problems, specialized architectures began to replace general-purpose serial-processing systems.

TRYING TO KEEP UP

Once again, but now for BI demands, the human desire for solutions to complex problems is growing faster than the exponential growth in processor speed.

The amount of data, sophisticated users, and software that generate complex, computation-intensive questions is growing. Numerous banks now have customer data warehouses sized in the 10s of terabytes (TBs), and telecom databases are in the 100TB range. Yahoo collects 1TB of data every four days, and Visa stores one petabyte (PB) of customer transaction data in five years.

As a result, parallel architectures are increasingly part of the solution choice. With 4- to 64-bit processors, these readily available servers can be clustered into any number of communicating nodes, with RAM and disc drives attachable in a wide range of configurations.