The Soft Underbelly

Data integration middleware: Will it become IT's second brain?

By David Stodder

Some time back, while browsing in a local bookstore, I came upon The Second Brain, by Michael D. Gershon, M.D. (HarperCollins, 1998). The author explores pretty much everything that happens once the "first brain" decides to ingest pepperoni pizza, eggplant parmigiano, egg rolls, salsa and chips, or whatever else stokes the fires in the belly (particularly in the wee hours). The book features chapters such as "Beyond the Teeth: The Domain Stalked by Heartburn and Ulcer," "Onward and Downward," and "Immigrants and the Lower East Colon." The author seemed to have wisdom to impart. I bought it.

"No poet would ever write an ode to the intestine," Gershon observes. However, "the ugly gut," he states, "is the only organ that contains an intrinsic nervous system that is able to mediate reflexes in the complete absence of input from the brain or spinal cord." He continues: "The brain in the bowel has evolved in pace with the brain in the head. Our enteric nervous system is not even small. There are more than a hundred million nerve cells in the human small intestine, a number roughly equal to the number of nerve cells in the spinal cord." Describing the "vast chemical warehouse" that is the gut, he declares that "neuroscientists, whose horizon ends at the holes in the skull, are continually amazed to find that the structure and component cells of the enteric nervous system are more akin to those of the brain than to those of any other peripheral organ."

While ingesting some steamed pepperoni pizza at the recent Anaheim, Calif. gathering of The Data Warehouse Institute (TDWI, August 12-17), I thought of this book. Not because of the pizza: The inspiration was all the dialogue at the conference about data transformation layers, metadata integration, and middleware. As cochair of the Best Practices Awards track, I had the opportunity to hear a variety of user presentations - companies that had to overcome latency, performance, enterprise modeling, reporting, and other challenges to achieve data warehousing success. However, one overriding similarity was the amount of brainpower devoted to each one's middleware for extract, transformation, loading, integration, and metadata.

To my mind, what made the winner of this year's TDWI Best Practices Award - eBay - most remarkable was its innovative data staging and transformation layer. As we move from passive to more active, sense-and-respond style business intelligence (BI) applications that are embedded in distributed systems, IT's chief role in an enterprise's business success could rest on the power and intellect of this "second brain": that is, the middleware infrastructure.

Data integration middleware exists because people (or applications) need to share data. Historically, as most data administrators and metadata architects are only too aware, few systems were designed with data sharing as a chief objective. However, as collaborative e-business relationships proliferate - and as mergers and consolidations sweep industry after industry - data sharing becomes more critical to IT success. Moments after we ingest pizza, all the specialized "applications" in our guts begin sharing information to make the necessary chemical transformations possible. Decisions are made, and the gray matter between our ears has little to do with it. Could middleware be headed in a similar evolutionary direction?

Bridging Old and New

Gershon describes the enteric nervous system as a "curiosity, a remnant of our evolutionary past that has been retained." I suppose the same could be said for integration middleware, considering its chief role is often that of moving data or connecting business processes between old ("legacy") and new systems. Like the gut, middleware operates best when it is least noticed, doing its "chemical" transformations and routing data where it needs to go to fulfill business processes or the data access demands of decision makers. However, as databases and applications have grown in size, number, and complexity, so too has the middleware solution portfolio.

As with enterprise application integration (EAI) solutions, the hub-and-spoke architecture has long held mindshare as the most promising alternative to the nest of point-to-point connections between interfaces and data sources that ensnare most large enterprises. These connections - and the multitude of reports usually spawned from them - contribute massively to data quality problems. And typically, the "brains" that created the point-to-point interfaces have long ago walked out the door, leaving behind little or no documentation. By establishing a "hub" integration engine that uses "spokes," or adapters to connect to each of the data sources, companies can centrally manage integration.

Unfortunately, performance problems have dogged these newer solutions. Some have been slow to take advantage of critical technology advances, such as multithreading, parallelism, and clusters. With return-on-investment and cost-of-ownership priorities dominating the frontal lobes of today's CIOs, there's less and less patience for solutions that create expensive problems that themselves require an army of systems integrators to manage. Today's silver bullet, XML, could allow companies to create more nimble distributed architectures that don't suffer from the performance woes of centrally managed systems. However, XML, while bringing us closer to the nirvana of intelligent, universal data integration, is only a piece of the puzzle.

The Vendor Scramble

Homegrown solutions remain the chief competition for an increasing pool of vendors, which now includes EAI players that have added data integration to their arsenals. The major DBMS vendors are there, too: but like the front-end BI vendors, they have their own technical and business agendas that don't always lead to easy universal data integration. Independent providers are rising, including Ascential Software, a long-time player that now has a war chest - thanks to its sale of the Informix database software to IBM - with which to grow as a dominant data integration and media asset management solution provider. Most likely, we will see continuing consolidation as the major software houses bulk up with XML, metadata management, loading, and other tools - and try to figure out whether to confront or avoid Microsoft.

One of the more interesting vendors I've met with this year is Data Junction, out of Austin, Texas. Quietly prospering while the more famous (and expensive) B2B integration players rose and fell over the past year, the company is on the verge of expanding from its data transformation and mapping niche to include event-driven process flow integration. The company is naturally suspicious of "central planning" solutions that sound great but end up leaving IT departments with more to manage, not less. Data Junction tries to come in with smaller footprints that enable customers to avoid trading flexibility for stronger integration. "Tactical victories are not dead ends," says Mike Hoskins, president of the company.

Rate This Article Rating: 1 (best) 2 3 4 5 (worst) Comments: Optional e-mail address:

At TDWI, it was clear that Informatica has emerged as the leader. The company's data integration platform, successful on its own, has become an integral part of other vendors' packaged application solutions. But as Informatica moves aggressively to become a more complete provider of analytic applications, BI competitors are formulating competitive strategies. One point of attack has been to expose what they view as performance and cost issues associated with Informatica's server architecture.

SAS Institute, for example, is working on the theme of "doing more with less" and taking a distributed resource management approach to integration performance. The company has begun reselling Platform Computing's LSF Job Scheduler, a heterogeneous load balancer, with its SAS/Warehouse Administrator release 2.2. The solution aims to help organizations move toward more automated, event-driven, self-monitoring data transformation and integration that is sensitive to underlying server capabilities. SAS expects to announce other data warehouse administration solutions in the coming months aimed at accelerating ETL layer performance without adding new layers to manage.

As I listened to the Best Practices Awards winners, it seemed that many are looking for just that: self-managing middleware that is independent of whatever BI tools users desire for reporting, analysis, and other activities. Perhaps they will someday have that second brain, deep in the bowels, which, in a perfect world, takes care of business without fanfare - or too many antacids.

David Stodder [dstodder@cmp.com] is editorial director of Intelligent Enterprise.