A Discovery With No Name

KDD-2000 reveals data mining as the secret ingredient behind e-CRM

Through a window blurred by steam and smudge, fishing boats rocked beside the pier. Others glided by purposefully, headed out to sea. Deep as we are inside the information explosion - determined as we are to deliver knowledge and intelligence - we still need places where little information is necessary. Thanks to Diane Cerra and Dorian Pyle, I found myself at No Name Restaurant, just one pier down from Boston's World Trade Center, which was hosting the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD-2000; August 20 through 23).

Established in 1917 and still owned by the same family, No Name operates from a simple, frayed menu: no fancy descriptions, just a list of items like "Broiled Bluefish," "Fried Scrod Sandwich," and "Boiled Lobster." "If you have any complaints, just ask to speak to the management," said the burly waiter, who might have been the management. We had no complaints. Based on my one and only visit, I doubt that this legendary Boston seafood restaurant will ever need personalization, customer relationship management (CRM), or any other information-rich "bait" with which to capture and keep customers. As long as the kitchen keeps producing what it has for 83 years, it won't even need a fancy name.

Alas, in the Internet Age, few enterprises would dare compete with such anti-marketing tactics. Amid a sea of information, B2B and B2C concerns are racing to apply scientific methods to discover which actions and messages draw customers. The annual KDD conference focuses on the science of knowledge discovery: Not surprisingly, the hottest industry topic at KDD-2000 was CRM. Undergoing a metamorphosis from mere sales-force automation tools, CRM is increasingly concerned with the intelligent use of data resources about customers and markets, and linking activity to supply chain management. State-of-the-art CRM software solutions must include an analytic application: that is, an integrated package of data transformation, data warehouse/mart, and business intelligence tools. For intelligent CRM, analytics are hardly side dishes like rice pilaf or coleslaw: Analytics produce the glue that binds together the single view of the customer - and from the customer's point of view, the single view of the enterprise.

Some analytic applications include data mining algorithms for pattern matching and visualization. According to a recent IDC report, this industry development helped to blunt the growth of data mining tools as a separate market sector. However, most analytic applications still deliver ROI through data integration and transformation to support traditional extended spreadsheet activities such as OLAP query, reporting, and analysis. While analytic applications have pushed business intelligence (BI) out to a new category of customer-facing business users, more ambitious data mining activity is still left to statistical analysts and other specialists.

Data mining, as defined by Michael Berry and Gordon Linoff (in Data Mining Techniques, John Wiley & Sons, 1997), is "the process of exploration and analysis, by automatic or semiautomatic means, of large quantities of data in order to discover meaningful patterns and rules." Data mining is an umbrella category covering a range of algorithms and techniques used to answer questions that have many variables, such as "Who are my most valuable customers?" Ideally, data mining fits into a knowledge discovery process that includes OLAP-style aggregation, segmentation, and dimensional modeling. According to IDC, SAS Institute is the clear data mining tool suite market leader, followed by SPSS, IBM, and Silicon Graphics. Other vendors such as Cognos, MicroStrategy, and Mathsoft have bundled significant data mining features with their packages.

The E-CRM Effect

Vendors of database marketing automation solutions have long partnered with SAS and other BI suite vendors: but the heat on e-CRM could reorganize the market. Advertising agencies are under pressure to become quantitative - ultimately, perhaps, to fuse their content development with personalized database marketing across multiple channels. Agencies have been scooping up data mining software and consulting concerns. For these talents, they are competing with vendors of e-BI suites, search engines, and database software. Earlier this year, Vignette acquired DataSage, enabling the vendor to bundle data analysis and mining into its e-business application suite. In my last column I mentioned Blue Martini, which is similarly developing sophisticated tools to go with its Customer Interaction System suite. Database marketing solution vendors, such as Xchange, are hardly standing on the sidelines; earlier this year, Xchange acquired Knowledge Stream Partners, a leading data mining firm, for $52 million.

Also competing for data mining talent are e-tailers, such as Amazon.com, and all sorts of large click-and-mortar businesses that are dying to analyze their terabytes of data, clickstream or otherwise. Amazon had a booth at KDD-2000, and you could see representatives from other businesses trolling the talent at the conference. KDD-2000 entertained nearly 1,000 attendees, beyond projections for this academically oriented conference. An industrial track featured Michael Saylor of MicroStrategy, James Goodnight of SAS, and various users and vendors speaking on real-world applications. The conference also hosted panels on personalization, privacy, and tool integration standards. It was refreshing to hear these issues presented and debated without the omnipresent marketing flavor that dominates today's industry trade show conferences.

Listening to several of these presentations, it was clear that as opposed to traditional BI activities, which excel at delivering a picture of the current state, data mining is a catalyst for business change. This is especially true in an e-CRM context. Data mining activities often begin with questions that go to the heart of business objectives: What kind of customers do we want? How effective are we at retaining them and getting more of their business? And most dramatically, how might we alter our behavior and structure to capture new market opportunities?

If we've learned anything from the data mining industry's brief history, it is that tools are not silver bullets. In other words, the tools can only facilitate building predictive models, discovering patterns, and finding affinities. The hard work will always involve human intelligence - which certainly might someday be embodied in autonomous agents, robotic sensors, or embedded software. But that's for research folks to figure out: To exploit the realtime potential of e-CRM, many companies are prepared to throw caution to the wind in the race to link customer data discovery with action in the form of instant offers, one-to-one marketing, and personalized services.

Matthew Cutler, co-founder of NetGenesis, gave a strong presentation that described not only the potential of intelligent e-CRM, but also the absolute necessity of developing performance metrics to evaluate the results. If we are ever to move away from bludgeoning ourselves with spam mail and other forms of media saturation, marketing professionals will need sophisticated evaluation software. Paul B. Chou, Edna Grossman, Dimitrios Gunopulos, and Pasumarti Kamesam, representing IBM T. J. Watson Research Center and the University of California, Riverside's Department of Computer Science and Engineering, presented a paper that provided a deeper discussion of the data mining process for customer prospecting, including profiling and clustering techniques. Their paper showed how the ultimate goal of data mining in an e-CRM framework is to build a model that best predicts the outcomes, thereby sharpening marketing activity.

Unearthing Personality

Data mining is perhaps most essential for large-scale personalization. While it is nearly impossible for journalists, like me, to get user organizations to talk about such strategic and sensitive stuff (given today's privacy debates), we can easily see the end results on the Web or in our email inboxes. KDD-2000 featured several presentations on algorithms for sharpening collaborative filtering and cross-sell recommendations; Brendan Kitts, David Freed, and Martin Vrieze of Vignette presented a particularly insightful paper describing their method.

Search engines, which both touch and amass huge data sets, are becoming infused with data mining. Gary Flake, Steve Lawrence, and C. Lee Giles of NEC Research Institute offered a paper describing how algorithms embodied in a Web crawler could uncover natural Web communities based on hubs of hyperlinked documents and structures. It was easy to see how such a capability could lead not only to more personalized search engines, but also to Web sniffers that could draw conclusions based on individual customers' clickstreams. Privacy watchdogs might have something to say about that.

Is e-CRM the whole story? Will "data mining" cease to exist, except as the secret ingredient in sophisticated e-CRM solutions? Hardly: At KDD-2000, it was clear that some of the most exciting applications of data mining algorithms are in financial services and the emerging field of bioinformatics. However, software development follows opportunity: Right now, e-CRM is data mining's mother lode.