Companies want to make better use of existing resources, and IT vendors are finally listening. After the IT spending flameout of the last several years, two sectors are emerging from the ashes: integration and business intelligence (BI). This year, our product reviewers discuss how these increasingly connected technologies are changing the competitive landscape.

According to Greg James and Jack Hakim, data mining and analytics continue to grow and evolve into new areas of the business, with integration playing a major role. Nelson King singles out Borland IDEs as better for supporting development across multiple platforms. Rajan Chandras discusses how customer data integration applications are hitting a sweet spot with companies. Finally, the increased demand for resources to support these additional data loads is moving grid computing — integration of computing resources — into the mainstream, according to Ganesh Variar.

Data Mining Evolutions

The data mining sector was certainly not spared from the recent economic woes that cast a pall over the entire IT industry. Behind the scenes, however, a great deal of progress was made. After attending five data-mining conferences in 2003, I'm struck by the steady adoption of data mining tools and methods in all sectors. Many case studies of "routine" data mining implementations showed significant contributions to organizations.

I also perceive a developing schism between the academic and commercial data mining communities. Researchers at the KDD-2003 Conference complained that industry isn't providing enough access to real-world data for research and development. At the same time, papers and attendance by industry professionals continue to decline. This was not the case prior to 2001.

The major commercial suites offered by Angoss, Insightful, Megaputer, SAS, and SPSS are now mature, fully featured, robust systems. They all offer refined GUIs, a plethora of modeling and visualization nodes, excellent integration with DBMSs, client/server architectures, and impressive performance.

On the database front, all three leading vendors — IBM, Microsoft, and Oracle — are well into their second generation of integrating data mining within their DBMSs. They are now busy developing enhancements to their engines to run data mining models directly within their databases.

Predictive model markup language (PMML) is evolving into the de-facto glue that makes all this possible. Seamless integration between modeling tools and database engines will bring significant productivity gains. Soon, you will no longer have to manually stage the data on a workstation or server to build and tune models. And you'll no longer have to manually generate and deploy scoring programs.

Finally, this year the community saw adverse publicity as data mining was vilified through association with the U.S. Government's Total Information Awareness (TIA) program. The Congress decided TIA was too controversial and a potential source of R&D funds disappeared as soon as it appeared. Nevertheless, the first commercially available text-mining packages emerged this year, and their potential market is easily as large as the one for data analysis.

Greg James [greg.james@nationalcity.com] is a vice president of National City Corp. and manager of the Retail Marketing Quantitative Methods group. He also teaches university computer science and data mining classes.

Analytics Everywhere

Over the next few years, business analytics will become a pervasive element of enterprise computing platforms. There are several business as well as awareness factors pushing this trend: Information is growing faster than Moore's law (in some industries it's doubling every three months); without BI and data mining, organizations can't leverage the data to make better decisions.

Analytics are converging with the transaction system. Microsoft will have a substantial analytic capability in its next version of SQL Server. Oracle 10G will include a substantial upgrade of Oracle analytic services that works against transactional data.

Analytics are also becoming a critical underlying part of BI. BI vendors are expanding the ability to leverage more sophisticated analytics under the covers, enabling skilled business analysts to create specialized guides and reports that they can distribute broadly among corporate analysts and managers.

As IT continues to move away from being a cost center, analytics are becoming a key enabling component for how IT can create value. Analytics are already critical in many departments, providing many functions such as forecasting, modeling, and decision-making.

Microsoft has identified BI as one of its six key initiatives. Other vendors, including IBM, Oracle, SAP, and Siebel Systems, have all indicated that they are moving to integrate BI capabilities throughout their platforms. Lower-cost BI sitting on top of analytics will become a part of the infrastructure and will reduce the cost and increase the value and availability of analytic capabilities across the organization.

Vendors looking for an opportunity to provide more value and services to their clients have initiated significant BI efforts premised on the value that pervasive non-stop analytics will provide. Many early adopters of BI have realized significant ROI. Having a horizontal BI capability in a horizontal infrastructure is a logical component of the FUSION model that Gartner believes is the dominant IT trend; its conviction will further convince companies to incorporate BI and analytics as a horizontal service, making it far more ubiquitous across the organization.

Jack Hakim [jhakim@ecwise.com] is a principal at EC Wise Inc., with offices in the San Francisco Bay area as well as Seattle.

In Pursuit Of Customer Data Integration

Understanding the customer has never been so easy — or so difficult. On one hand, we have unprecedented information about our customers; on the other hand, weaving a coordinated tapestry from this collage remains a significant challenge. Despite various efforts at meeting this challenge — including CRM, data warehousing, and enterprise application integration (EAI) solutions — presenting a single meaningful customer definition continues to be an elusive goal.

Now, fueled by venture capital and armed with remarkably similar offerings, vendors such as DWL, Journee, MetaMatrix, and Siperian are eager to help you create this single customer definition.

As I understand it, the typical solution comprises a central J2EE-based repository that holds metadata and reference data (customer keys) together with rules governing the data, which connects to data sources using application/data adapters. The repository creates a logical unified customer model with persistent customer data, at least partly. This "single virtual customer" can now be of use to CRM solutions, OLTP/Web applications, and data warehousing/BI solutions. Additionally, the generic nature of the solution provides vendors with the agility to package it vertically (for industries) or horizontally (for entities other than customer).

To reach out into the marketplace — and survive competition from SAP and Siebel Systems, which have their own data integration solutions — these vendors are partnering with systems integrators of proven delivery capabilities (Journee with Computer Sciences Corp. [CSC] and Siperian with Accenture).

The greatest weakness for these vendors may be their equity model; their best chance of survival may lie in eventual merger with other "data heavyweights" such as Informatica and Ascential. And I do hope that they survive; frankly, I like their offerings. And if you're in the market for data integration, I think you will, too.

My acknowledgments to DWL, Journee, and Siperian for providing information beyond their Web sites, and to Rob Beauchamp (CTO of Journee) and Ken Huong (CTO of Siperian) for their help.

Rajan Chandras [rchandras@hotmail.com] is a principal consultant with the New York offices of CSC Consulting (www.csc.com). The opinions expressed here are his own.