Beneath the Waterline

Most users are blissfully unaware that data integration poses increasingly complex challenges

By Philip Russom

Continued from Page 1

Integration Trends

Toward packaged integration solutions. Much of the success of integration products can be attributed to the fact that it's easier for IT to buy an integration product and configure it to work with existing systems than it is to build an in-house integration system from scratch. Software vendors are aware of this market requirement and produce high-level "connectors" that plug and play easily with common packaged applications for ERP and CRM. For example, prepackaged data integration connectors to SAP and PeopleSoft products have been around for years (from Acta Inc., Ascential Software Inc., Informatica Corp., and so forth), whereas connectors to Siebel Systems Inc.'s CRM platform have appeared more recently.

Midtier caches for data integration. Many IT departments design and build their own data integration platform in the guise of the operational data store (ODS). It is essentially a database (built with IT's preferred DBMS), conceptually located in the middle tier of a distributed architecture. An ODS usually contains one type of data (customer records and outstanding orders are popular topics), stored in simple tabular structures optimized for fast access. Hence, an ODS is a midtier data cache that adds value to data by integrating it and by providing high-performance access.

Although most ODSs are "homegrown," commercial versions for popular cache topics are emerging. For instance, eCommerce Cache from Acta Technology is a packaged data integration solution that employs an ODS with a rich data model that connects front-office e-commerce applications to back-office systems.

From ETL to EAI. Data warehousing professionals have long depended on tools for extraction, transformation, and loading (ETL). These tools and their best practices are now - at least, in a few companies - being applied to general data movement and integration tasks. ETL tools offer several advantages over general data integration tools, such as complex transformations, aggregate building, flexible work flow, and sophisticated business rules. Most ETL tools are designed around a hub architecture that enables the consolidation of data from multiple sources in a way that application integration technologies cannot. However, ETL tools assume considerable data latency.

"Instead of just the data warehouse, ETL vendors are trying to take on the whole enterprise," says Mike Schiff, vice president of e-business and business intelligence at Current Analysis. "They want their data integration products to become a fundamental part of EAI infrastructure so they can reach a broader market. But to move data from anywhere to anywhere - not just into the warehouse - you need more than ETL technology. You need a realtime strategy."

That's why recent releases of major ETL tools - namely, Acta's ActaWorks, Ascential's DataStage, and Informatica's PowerCenter - support IBM's MQSeries, the unchallenged market-leading platform for realtime messaging. Again, product offerings are progressively combining complementary technologies from both realtime application integration and latent data integration.

XML enables B2B integration. One of the barriers to data integration in B2B environments is that each partnering company may have a unique data format for information exchange. XML-based solutions (whether homegrown or vendor produced) are quickly becoming the preferred method for mapping business transactions and documents from one format to another. XML is ideal for this chore because it's easy to implement (being both machine and human readable); it is supported by a growing number of tools; XML-based standards for describing B2B data are finally taking hold; and it's designed for information exchange over the Internet.

B2B integration requires both latent data integration and realtime application integration. For instance, InfoShark Inc. is a data integration vendor that provides an economical XML-based platform. As the company shifted its focus to B2B integration, it discovered that its customers also needed realtime messaging based on XML, which led it to acquire WebXi Inc.

Data integration scalability. Two strong currents are sweeping data integration technology toward a scalability crisis. On one hand, online transactions, data volumes, and user communities craving data access increase daily. On the other hand, data integration gets harder because of shrinking batch windows, increasingly complex distributed computing environments, and rising expectations for performance.

The Future of Data Integration

Based on these trends, where is data integration going?

• Companies will continue to integrate new information sources like Web server logs and supply chain management applications. As market demand arises, vendors will produce packaged data integration connectors and data models for these sources.
• Although most IT integration budgets to date have been spent on customer data and internal issues, the pendulum is swinging toward B2B tasks. A data integration strategy must include XML support, as well as functionality focused on product catalogs, procurement documents, and other typical B2B data types.
• As XML spreads, data integration practices and tools must generate, parse, store, and query XML data natively, as well as map to and from multiple XML-based standards.
• As computing environments become even more distributed (driven by the adoption of Web architectures), IT departments should build more ODSs as midtier data integration platforms that collect the information users want, provide a data-oriented approach to application integration, and contribute to scalable performance.

Rate This Article Rating: 1 (best) 2 3 4 5 (worst) Comments: Optional e-mail address:

Turning to the broader EAI space, the need for integration technologies is increasing as corporations adopt the business practices of e-business, e-commerce, and B2B online trading. It's important to note that this need encompasses all types of integration technologies and that a single company needs multiple types. To address the market demand, some vendors are assembling (through partnerships, development, or acquisitions) a kind of "universal integration platform."

By supporting multiple integration technologies, a vendor can appeal to diverse customers and prospects across a wide range of industries and business situations. By adopting a universal integration platform, IT personnel avoid the risky task of "integrating the integrations" that a multivendor approach entails. Although leading vendors like BEA System Inc. and Vitria Technology Inc. are moving aggressively to satisfy this trend, the universal integration platform is still a new concept, so it will be a while before credible offerings appear.

Philip Russom, Ph.D. [www.PhilipRussom.com] is a Giga Research Director at Forrester Research Inc., where he provides advice to user organizations about business intelligence, data warehousing, and data integration.