Beneath the Waterline

Most users are blissfully unaware that data integration poses increasingly complex challenges

By Philip Russom

Where is data integration going? Like other enterprise application integration (EAI) segments in recent years, the practice of data integration and, therefore, the tools software vendors offer for it have seen many changes. For example, the traditional EAI space has evolved with the tremendous growth of e-business and B2B transactions; and in response, the data integration segment is using XML-based solutions, ETL tools, and midtier caches. As your company grows, you will

inevitably incorporate new sources of information. You need insight into the best strategies and technologies for integrating the diverse data types that may lie in your future.

In this article, I'll identify the prominent trends that are wending their way through data integration practices and tools and put them in a framework that can help you understand where data integration is headed today.

Before we can begin to understand the trends, however, we need to define data integration relative to other integration technologies.

Integration Technology Differences

Integration is like an iceberg. Your end users see the benefits of information access - the tip of the iceberg - but most are blissfully unaware of the sophisticated integration technologies (what's "underwater") required to achieve the benefit.

"Beneath the water, integration needs to enable ubiquitous data access, shared application services, and robust business process management," says Tyler McDaniel, senior analyst of application strategies at Hurwitz Group Inc. "These [areas] are the traditional EAI segments, although the [B2B] phenomenon is raising the stakes for each of them." Hence, the longstanding segments of the EAI space - namely, data integration, application integration, and process integration - are now being joined by solution-specific categories, such as B2B integration.

But what's the role of data integration relative to other integration technologies? The quadrant chart in Figure 1 answers this question. It identifies two key attributes of integration technologies and draws them as opposing axes, which, in turn, provide a context for plotting integration technologies relative to the attributes.

Realtime vs. Latent Integration

The primary attribute of an integration technology concerns how quickly it can move information. At one extreme, your users need information integrated between applications in real time - or as close as is possible. For instance, when customers in arrears pay their outstanding debts, the billing application reflects these changes, but the help desk application should also reflect these changes immediately, so support personnel do not deny service if the customers call. At the other extreme, your users don't need second-by-second updates to databases that support reporting, customer-base segmentation, direct marketing, and other business tasks where data "freshness" is not a requirement.

For realtime integration, IT personnel generally turn to application integration technologies like object request brokers (ORBs), queues, and other messaging services. When users can tolerate considerable "latency" - that is, a lag in time from when data is created or changed until it is fully propagated to other systems - data integration is an appropriate choice. It relies on database technologies like gateways, metadata, queries, data set transfers, and bulk data loading.

Transactional Messages Or Bulk Data Sets?

Another important attribute of an integration technology concerns the size and type of data structures it is capable of moving. Again, the attribute reveals differences between application and data integration. A generalization that applies to most cases is that application integration moves many small messages in a transactional methodology. Because one of its goals is realtime information movement, application integration tends to move data as soon as it is created or changed. By comparison, data integration is more appropriate to moving relatively small numbers of large data sets (in the multimegabyte and gigabyte range). Latency is tolerated, so regular delays in data movement occur until there is an off-peak batch window.

Despite their differences, the line between application integration and data integration is not always easy to draw according to these attributes. For instance, a message queue for application integration can be configured to delay messages until an advantageous window opens. And a batch process for data integration can run every few seconds. (The overlapping ovals in Figure 1 represent this intersection.)

Process Integration and B2B Integration

But there's more to integration than applications and data. Many companies realize that their real goal is to integrate business processes, both across the enterprise and beyond to customers and partners. For them, process integration (sometimes called business process management or business process integration) is the primary technology. And packaged solutions focused on B2B integration are a lucrative - and, therefore, very active - segment in EAI at the moment.

In terms of the quadrant chart in Figure 1, process integration is mostly an offline modeling exercise that relies on other tools to do the actual data movement, which makes it very latency tolerant. B2B integration is about transactions processed in real time. Yet these transactions tend to include contractual business documents (such as invoices, requisitions, or RFPs), which makes the size of the transactional message relatively large. Hence, the realtime, relatively large messages of B2B integration place it in the upper-left quadrant of Figure 1.

Now that we have defined our terms and created a framework for understanding the context of integration categories in the EAI space, we can turn our focus to data integration and its current trends.

Integration Categories Are Merging

As if the lines between segments of the EAI marketplace weren't fuzzy enough, an ongoing trend - driven by user need - is further blurring the lines. IT departments that need one type of integration technology, tend to need other types as well. In fact, the most commonly needed pair is application integration and data integration. In response, software vendors, who had previously focused on one category of integration, have scrambled to broaden their product offerings to include multiple categories.

Vendor partnerships - many of them "co-opetitive" - are common in EAI. For instance, IBM's willingness to partner with a wide variety of integration vendors has helped make MQSeries ubiquitous as a messaging platform for realtime integration. Another example is Excelon Corp. (a B2B integration vendor), which recently struck a reseller agreement with Information Builders Inc. (for data integration capabilities).

Partnering is quick, but many integration vendors prefer to build their own functionality so they control its content and release schedule. For instance, many data integration vendors are aggressively developing support for XML, which eventually will be the basis for packaged B2B integration solutions.

Mergers and acquisitions broaden EAI product offerings to include more types of integration technologies. DataMirror Corp. and Constellar Corp. are a classic case of complementary vendors merging. The combined product offering now includes the DataMirror Transformation Server (a peer-to-peer middleware approach to realtime data integration) and Constellar Hub (a hub-and-spoke architecture for complex calculations and aggregations of large data sets). As further examples, WRQ Inc. (data integration) acquired SuperNova Enterprises (application integration), and Level 8 Systems Inc. (application integration) acquired Template Software Inc. (process integration).