Mass Movement

Scalable data integration is crucial as businesses face increasing data volumes and transactions

By Philip Russom

Executive Summary

Philip Russom Ongoing trends are driving up data volumes and making data integration increasingly difficult to perform. If your company moves and integrates a gigabyte or more of data daily, these escalating trends will soon lead to a data integration crisis that could adversely affect your important business processes. Therefore, to achieve scalable data integration under extreme conditions, your IT organization should adopt techniques such as parallelism, as well as consider simply adding more hardware.

How do you define scalability? The definition depends on what you're trying to achieve with your IT systems. For many, the top priority for scalability is high-speed processing that enables great numbers of transactions per second. For others, the primary need is a system that can scale up to large user counts or voluminous data storage. However, in the world of data integration, scalability means massive data throughput. You may be facing a data integration crisis sooner than you think.

Today, two groups of trends are driving the business demand for data integration and the challenges to its implementation. (See Figure 1.) On the one hand, data volumes escalate daily because of increasing numbers of online transactions, applications needing integration, and users who demand access to integrated data. On the other hand, data integration becomes harder to implement because of shrinking batch windows, increasingly complex distributed computing environments, the need to preserve network bandwidth for applications, and rising expectations for performance.

As these trends move forward, they will drive many corporations toward an impending crisis of data integration scalability. In this article, I examine these trends as well as the technologies and IT practices that can help you prepare for a possible data integration crisis.

Trends Driving Data Integration Volume

The greatest challenge to data integration scalability is the explosion of data volume that almost all companies are currently experiencing. But similar challenges come from escalating numbers of transactions, applications, and users. (These trends appear on the vertical axis in Figure 1.)

Burgeoning data volumes. Many corporations already move gigabytes of data daily. This movement is part of a data-oriented strategy for integrating internal applications and transacting with business partners, as well as for loading decision-making data into data warehouses, corporate portals, and knowledge management systems. As companies operate more and more like e-businesses over the next few years, the daily volume of data movement will increase to hundreds of gigabytes, pushing the scalability of data integration technologies to their limits.

More transactions online. Direct-to-consumer online transactions are increasing steadily, although not as quickly as numerous failed B2C e-tailers had hoped. The most dramatic increase, however, concerns B2B trade. Market research firm Gartner Group forecasts that B2B online transactions will amount to $6 trillion in 2004. Dollar value aside, the number of transactions will increase correspondingly. Because B2B online transactions typically rely on data integration technology, scalability will be an important issue.

Increasing numbers of applications. The steady "digitization" of corporate processes - a long-running trend accelerated by e-business - brings new applications online regularly. These applications must integrate with existing systems so they can contribute to "information synergy" across the enterprise or else they become nonleveraged silos. For instance, data integration tools and best practices typically link newer front-end applications (perhaps for e-commerce or other customer-facing tasks) with older back-office systems (for shipping, billing, inventory, and so forth). The IT organization in any company - large or small - must cope with an ever-expanding list of applications that require some level of integration.

Growing user communities. E-business is about performing business processes online, which improves efficiency and shares information broadly for better decision making. As the practice of e-business rises, the number of users logged into applications increases as well. And a large portion of these users need to access integrated data in corporate portals, data warehouses, and caches of cross-channel customer or partner data. IT in the Fortune 1000 today is already supporting thousands of users. As user communities grow, so will their demands for pre-integrated data.