Intelligent Enterprise

Better Insight for Business Decisions

RSS
Webcasts
Digital Library
Subscribe
Home

April 16, 2002

Behavior: The Next Marquee Application

It's time to start using more advanced analytics

By Ralph Kimball

The focus of data warehousing has evolved along a steady and inexorable progression in the past 20 years. Competitive pressures and new generations of management have brought us over time to the threshold of a new era of analytics.

The 1980s and '90s each had their distinctive "marquee" applications. In the early '80s, a 50MB database was pretty large. But, we were happy to be able to analyze the basic sales numbers of an organization. The marquee data-warehousing application of the '80s was shipments and share. We were delighted to see how much product we shipped each month and, if we were lucky, we could see what fraction of the total market that represented. In a sense, these early data warehousing applications represented the first time we could drill down from the annual report to start analyzing the components of our businesses. In the '80s, our analytics were simple: this month vs. last month or this month vs. a year ago. And perhaps the most difficult calculation was our share of a market this month vs. the same share a year ago.

By the early '90s, our capacities, techniques, and analytic expectations had progressed beyond simple shipments and share numbers to demand a full analysis of profitability at an individual customer level. At the beginning of the '90s, the most sophisticated data warehouses were already analyzing revenue at the individual store or branch level. Certainly by the end of the '90s we were able to capture and store the most atomic transactions of our businesses in the data warehouse. The marquee data warehouse application of the '90s was customer profitability. We developed techniques for tying together the disparate revenue and cost data sources in order to assemble a complete view of profitability. The extreme atomic detail of the data allowed us to tag each transaction with the exact product and customer. In this way, we could roll up a full P&L perspective for each customer and product line.

Curiously, although the quantity of data available for analysis increased by at least a factor of 1,000 between the '80s and '90s, we didn't see a significant increase in the sophistication of our analytic techniques. We had our hands full just wrangling the huge databases. While there was some modest increase in the use of data mining techniques, these advanced analytic approaches remained a tiny fraction of the data warehouse marketplace. However, we did see a significant improvement in the ease of use of end-user tools for querying and reporting. The explicit SQL user interfaces of the '80s mercifully gave way in the '90s to much more powerful user interfaces for combining data from multiple sources, highlighting exceptions, and pivoting the data at the user's desktop to make the numbers jump out.

The slowness of adoption of advanced analytic techniques in the '90s, in my opinion, was also due to a cultural resistance. Business management has always been reluctant to trust something it doesn't really understand. I'll argue in a moment that we are finally ready for the cultural doors to open, and for advanced analytic techniques to be more visible and important, but we need to be patient.

When I began my data warehousing career in the late 1970s as the Xerox Star Workstation product manager, I remember that at least half of our prospective Fortune 500 clients really didn't use computers or numbers of any kind to manage their businesses. They managed literally by walking the aisles and by "gut feel." The transition over the past 20 years to an absolute demand for managing by the numbers is both the result of technology advances as well as a generational shift in the business world as younger managers arrive with computer training and familiarity. Thus, looking forward, we need to be patient as we wait for an even more analytic culture to assert itself. I'll also argue in the following section that the demands of the next marquee data warehousing application will force us to upgrade our analytic sophistication because this next marquee application is a lot more difficult.

CRM: The Stepping Stone to Behavior

At the end of the '90s, CRM emerged as an important new data warehousing application. CRM extended the notion of customer profitability to include understanding the complete customer relationship. Patricia Seybold captured the CRM perspective beautifully in her book Customers.Com (Random House, 1998). A data warehouse designer can read her book as a set of business requests that can be translated directly into a system design for a data warehouse. One of Seybold's most powerful points is the need to capture all the customer-facing processes in the business. Much of my own writing about conformed dimensions and the data warehouse bus architecture has been with the issues Seybold raised in my mind.

But CRM as implemented in the '90s is still just a transactional perspective on the customer. We count the number of times the customer visited our store or our Web site. We measure customer satisfaction by the ratio of successful product deliveries to the total or by the change in the number of complaints. Our analytic techniques are still the kinds of counts and comparisons we used for shipments and share calculations in the early '80s.

But marketing managers are constantly looking for new competitive angles. Many organizations now have a pretty good understanding of customer profitability. They know which customers turn out to be profitable. They know which campaigns yield the best customers. But marketing managers are thirsting for a deeper understanding of how to recognize and develop good customers, and conversely how to recognize and discourage bad customers. Marketing managers will get the next competitive edge when they can understand, predict, and influence individual customer behavior.