Intelligent Enterprise

Better Insight for Business Decisions

RSS
Webcasts
Whitepapers
Subscribe
Home

April 5, 2003

A Virtual Point of View

Despite limitations, virtual approaches to data integration hold great potential for certain applications

by Philip Russom

Traditional approaches to data integration copy masses of data and consolidate it in a physical database like an operational data store or data warehouse. As a complement to this tradition, a new practice (supported by several vendor products) seeks to avoid moving large data sets by modeling a virtual or federated database in the metadata layer of a platform for enterprise information integration (EII).

Virtual Views

EII is a relatively new type of integration platform that supports virtual views into multiple data sources. A view represents a business entity — a customer, a sales pipeline, or the performance of a manufacturer's production floor — in a metadata-based description. Applications access a view as if its data were physically located in a single database even though individual data may reside in a different source system. When an application accesses a view, the EII platform transparently handles connectivity with back-end databases and applications, along with related functions, such as security, data integrity, and query optimization.

The point of a virtual view is that it makes multiple data sources look like one. The so-called heterogeneous query — where one query fetches and consolidates data from two or more sources, each with its own data model — is beyond the capabilities of most applications and some query tools, because they don't have data integration capabilities built in. However, when an application accesses data through a view, it sees one database (albeit a virtual one), which simplifies data access and enables heterogeneous queries.

To create virtual views, IT personnel use an EII platform to extract metadata from source systems, then model views that map into enterprise data. The views can create new and innovative data structures that both integrate and repurpose data. Views are all about metadata, making them faster and easier to create and modify than physical databases, which are required with batch-oriented data integration methods. As metadata on steroids, EII is a flexible data integration technology that helps businesses keep pace with evolving data integration requirements.

The metadata-driven method of EII has a useful side effect. Older, batch-oriented, data-movement-intense forms — especially extract, transform, load (ETL) — move large data sets from data sources to target applications and databases "just in case" an application or user will need the data. However, most corporations move far more data than is ever accessed. And moving this excess data needlessly burdens source systems, networks, and target databases — not to mention wastes money.

EII takes the opposite approach: It only moves data when an application or user requests it, which helps reduce the loads on IT systems and the drain of network bandwidth that's inevitable when moving data. Furthermore, some kinds of data can't be copied across IT systems because of privacy issues. With EII, you can leave data in its original format on the system where it began and only perform data integration as needed.

Along these lines is the theory that an EII platform integrates data in real time. The reality is that an EII platform generates a lot of overhead as it connects to multiple databases, receives data from each, consolidates the data into a single result set, and passes a result to the querying application or tool. Depending on the query, this process can take several minutes, but possibly hours, which has led many people to dismiss EII platforms as too slow to be usable.