May 31, 2003

Data as it Happens

Data in real time, all the time, is what many enterprises want. To make it happen, IT needs to get the big picture — and not burn out on one-off solutions for single applications

by Mark Madsen

You can deliver data in real time from one application to another without too much effort. It's far more challenging to create a real-time data-delivery infrastructure that allows an enterprise to easily integrate applications on a repeatable basis, and yet is also flexible enough to accommodate change.

Unfortunately, IT has historically built or bought applications based on what specific business units required. While the systems often deliver the functionality the business units wanted, a great deal of time is spent tying together the menagerie of systems. Integration starts to take up a greater and greater share of IT's budget. And the integration is done piecemeal: Little thought is given to reuse, building the technology base, or establishing a forward-thinking infrastructure.

Many organizations bought into ERP systems thinking that as packaged suites they would solve the data integration problems. Many packages do provide real-time data integration, allowing systems to access data from within one module and make it available elsewhere in the ERP system. However, few businesses can run their entire organizations from within the bounds of a single ERP package.

CIOs and ERP managers of particularly large organizations are coming to a different view of data integration: not as an appendage, nor as a technology glue that you use to stick applications together, but as a strategic infrastructure that must be considered separately from ERP and other applications and systems. A real-time data integration infrastructure approach begins with this critical assumption.

However, few IT organizations are poised to fully reintegrate existing systems into a new framework, at least in the short term. Plus, such an infrastructure isn't something that a vendor alone can develop and package because the technology implementation varies so much from customer to customer. The responsibility falls on both the IT organization and line-of-business units, which must separate long- and short-term goals — but not lose focus on the compelling strategic value of an integration infrastructure.

Conceptual Models: A Beginning

The best way to begin sorting out something as complex as integration infrastructure is to work through some conceptually simple models and use them as analogies. Once you settle on a general model and its components, you're closer to selecting the technologies for implementation.

An interconnection model, or topology, is the highest-level model abstraction, offering a view of how systems will connect with each other. This topological view is important because it articulates how developers think about the organization's integration problems, and gives everyone a critical communication tool for discussions with business management.

In this section, I will discuss three conceptual models. Remember, conceptual model selection isn't about fitting the model with a particular set of applications and technology, which is unlikely to be perfect anyway. The goal is to settle on a model that best fits the objectives of your organization.

Model 1: Point-to-point topology. In this conceptual model, one application locates and requests data from another and sends it directly to that application (see Figure 1). This style is probably familiar, although perhaps not for real-time data integration. It looks like the traditional ad hoc approach, in which systems are linked in whatever way is most convenient at the time. However, "ad hoc" implies that there's no underlying design principle, which doesn't have to be the case with a point-to-point conceptual model.

A point-to-point architecture implies that you have a common, perhaps even standards-based way for applications to recognize and address each other, using common data transport methods and interfacing mechanisms. If chosen as the most suitable infrastructure, this architecture must be defined formally and allow multiple kinds of implementation, especially given the different applications and systems involved. Web services could emerge as the best way to implement point-to-point models; another key development could be peer-to-peer and grid technology, which could be adapted for the purposes that are the focus of this article.

Model 2: Information pipeline (or "bus"). Unlike the point-to-point model, this approach doesn't require applications to know where to send requests. Instead, all applications connect to a bus, which is common pipeline with a standard interface that dictates communication syntax and protocol (see Figure 2).

The primary difference between bus and point-to-point models concerns addressing. The bus makes data available without requiring applications (or developers) to determine where to obtain the data. We see this model used in the financial services industry due to the nature of financial information, how it's "published" to subscribers, and ultimately how it's used.

• Privacy
• Your California Privacy Rights
• Copyright © 2008 United Business Media LLC