Small information appliances, in large enough numbers, will generate massive data stores

Information Scavengers

Richard Winter                

A big change is coming in the high-end database scene, and I can tell you why. I can’t, however, tell you what that big change will be—not for a few months. Several concurrent changes in technology are afoot, the gravity of which has been seen before, that are bound to affect data and transaction volume in several industries.

The last time we saw an effect this big on very large databases (VLDBs), it was the early 1990s, when commodity economics and open systems converged to bring us parallel architecture. In addition to the implementation of parallel database architecture in open systems, the actions of relational database managers, the rise of the data warehouse concept, and other factors also enabled VLDB growth. But in my view, the revolutionary explosion resulted from the swift cost decreases of microprocessors and disk drives—which put VLDBs within reach for many new applications—and their use in parallel architectures.

Parallel architecture had been around for a while. What made it a major market phenomenon was commodity economics—the fact that microprocessors were so abundant and inexpensive compared to the proprietary hardware on which VLDBs were managed for the prior two decades. Now, here’s the connection not everyone makes: The PC sparked the revolution. Worldwide PC purchases drove those commodity economics. Hundreds of millions of PC sales resulted in cheap, powerful microprocessors and cheap disks—which, in turn, transformed VLDB economics.

Clearly, the PC phenomenon continues. Every day, more people buy PCs. And via the Internet, more people have access to large databases. Nonetheless, PC industry growth—although still the envy of most industries—is slowing. It seems the next revolution is coming from another direction: information appliances.

Information Appliances

IT analysts’ expectations are focusing lately on special-purpose information appliances. Examples include a handheld device with a wireless connection to the Internet, a phone that shows faxes and emails on one screen, a tiny gadget that health maintenance organizations will give chronically ill patients so doctors can check their vital statistics online, and a combination camera and handheld computer.

These devices have several interesting characteristics: Most of them are going to be even less expensive than PCs, they are targeted at specific consumer needs, many are small and mobile, and they are much easier to use and maintain than PCs.

Many people who do not have a PC or plan to buy one are expected to purchase these information appliances, as are many who already have a PC. From 1999 through 2002, IDC projects that about 56 million PCs will be installed in U.S. homes. Meanwhile, U.S. consumers are expected to buy an estimated 60 million information appliances. This figure is only slightly more than the number of home PCs, but the growth rate is quite a bit higher. Consumer information appliances are projected to bypass home PCs in annual unit sales in 2001 and lead them by about 25 percent by 2002. In other words, information appliances will be out there in extremely large numbers and increasing very rapidly, giving a new meaning—and an even more frenzied pace—to the effects of commodity economics.

Significance for Database Scalability

I see three implications for database scalability. First, many of these 60 million new devices will access databases. The 3Com Palm V, a member of probably the most successful appliance family to date, is already available with add-ons for wireless access to the Internet. This newest palmtop model comes with up to 4MB of memory and is capable of running a fairly substantial database application. That’s up from 1MB when I bought mine—about 18 months ago.

First of all, if Internet service providers are staying awake nights now wondering how to address extraordinarily fast and unpredictable growth in demand, what are they going to be doing in three years when tens of millions of additional consumers are on the Web with their information appliances? Most PC users will surely continue to use their desktops as well, so this army of information appliance-equipped consumers represents a demand in addition to the 150 million or so PCs (counting both home and business) already in use in the United States alone.

Second—and perhaps most significant from a database perspective—many of these devices are designed to gather information then forward it to the Internet for database storage. For example, with a wireless medical device that monitors patients with chronic conditions, the data goes to the doctor, but surely that data is stored in the patient record. A few applications like this could make today’s multiterabyte commercial transaction histories look puny. What if people have wireless cameras? Digital photos, of course, end up in a database. How about wireless digital audio recorders? Knowledge workers dictate their notes and emails and transmit them over the Internet, the speech is parsed by software such as IBM’s ViaVoice or Dragon’s Naturally Speaking and turned into text, and much of the voice as well as the text ends up in databases, too. All 60 million devices in the next three years won’t be pouring data into databases, it’s true. But, to paraphrase the late Illinois senator Everett Dirksen, with 10 million here and 10 million there, pretty soon it adds up to real data.

Third, some of these devices—per haps just 10 to 20 percent over the next three years—will contain databases. DBMS products already exist for the Palm products. And there are handheld devices with disks! IBM has developed a new “microdrive” that is half the size of a credit card and stores 340 megabytes of data. It will be introduced at $350 later this year but is expected to fall to a price near $100 within a few years. (In a 1996 article I wrote for Database Programming & Design, “Make Way for the Petabytes!,” I reported that such a device was predicted to appear, but the prediction admittedly sounded outlandish at the time. It sounds less outlandish now only because I have learned to suspend my beliefs about how much space and power you need for a megabyte.)

I’ll leave for a future column the question of how database professionals will cope with the prospect of tens of millions of mobile relational databases in the hands of consumers. If the databases are in information appliances, of course, they will be running dedicated applications. But just 10 million people walking around with 100MB of data each adds up to a lot of database management questions for me.

But Wait, There’s More

If I were selling Ginzu knives on late-night TV, this is when I would stop and ask, “And what would you pay for all these items?” Then I’d pause and say, “But there’s more!” And there is—the “more” has to do with the business implications of information appliances.

First, many of these projected 60 million devices will browse the Web, leaving behind a collective clickstream that will number in the trillions. That means an extraordinary fund of information will become available about the interests and habits of a group of consumers not currently on the Web for lack of PCs. Also, if the type of appliance is known, the clicks can be tied to a certain type of activity or interest. Much of that new clickstream will go straight into data warehouses so big that we can’t imagine how to build them today. But don’t worry, we have a few months to figure it out.

Second, there is TV. WebTV is already quite successful: Seven-hundred thousand users are out there, and their ranks are growing fast. And many other wrinkles and variations are about to emerge on the theme of combining TV, the Web, and other devices. One example is a device for the kitchen that combines a 9-inch TV, CD player, and Web browser. The amount of time Ameri-cans spend watching TV is about 10 times what they spend browsing the Web in a given period. Gadgets that combine the two technologies promise to increase Internet use and add to the ever more valuable clickstream.

Perhaps the ultimate commodity “appliance” is the smart card—a card with an implanted microprocessor. One use is as a subway pass, primarily for paying fares, but that obviously has the potential to generate information about riding patterns—information that can be warehoused and analyzed to improve the design and efficiency of the train network. Smart cards are already being used in other ways, and many more uses are still to come that will feed large volumes of data into even larger databases.

Now What?

As I warned you, this is one situation in which I know more about the “why” than the “what.” I don’t know exactly what is going to happen to large databases as a result of information appliances. I’m willing to bet on an enormous impact—well above and beyond the extraordinary impact we have seen from PCs in the last few years. I also expect a lot of new Web and database traffic—mainly in the form of new applications.

Beyond that, we’re probably talking about a more dynamic kind of information here. People using these appliances are not necessarily sitting at desks, working quietly with their PCs. They will be out, moving around the world, performing other activities, interacting with other people, and making decisions. If a million people across the country start doing something differently one morning because they saw something on TV and they want to know more about it—or buy it—then systems and organizations may have to react right away. And it isn’t necessarily going to be enough to record the activity, load it into the data warehouse overnight, and then discover what’s happening the next morning. The expected volume of the devices—along with their role as a more integral part of everyday life—suggests to me a new information dynamic in which extremely large volumes of information can pour in from all over and call for immediate action. This concept is very different from that of most information systems today and may well mark another major chapter in the use of VLDBs.

Richard Winter is a specialist in large data base technology and implementation, and is president of Boston-based Winter Corp. You can reach him via email at Richard.Winter@wintercorp.com or by fax at 617-338-4499.