The Medium Is the Message

Enterprise messaging offers useful alternatives for integrating enterprise applications

By Greg Barish

Today, companies are increasingly interested in how applications communicate with each other. Why? Because enterprises have already built applications that interface their underlying data stores; in the true spirit of B2B integration, they want to be able to connect their applications together. Certainly, companies are still interested in exchanging their valuable, underlying data with each other. But as software-based applications have evolved into sophisticated application servers, industry folks have realized that the business logic provided by the applications is itself valuable. By connecting applications together, enterprises can exchange both data and business logic. And by connecting Web-based applications together, this exchange can occur without building proprietary networks or relinquishing access to the databases behind them.

In bridging applications together, you must handle various communication paradigms. The most common paradigm that comes to mind is the client/server exchange, which involves the client sending information to the server (request) and the server returning information back to the client (reply). In this case, both the request and the reply contain meaningful data. However, another style of communication that exists is the broadcast paradigm - information "producers" send out information to "consumers" as necessary. What is interesting about this paradigm is that, typically, the only value is in the information going from producer to consumer (unlike client/server, where both request and reply are meaningful).

For example, think of applications that react to stock market changes. As security prices fluctuate throughout the day, these applications need to be notified. A poorly engineered approach to this problem would be to have all of the client applications continually query server applications or data sources. But this solution would lead to a tremendous amount of network traffic and is generally not scalable or reliable. Instead, it seems more natural to push this information out to these consuming applications as it changes, in an event-driven style. While many people are familiar with the client/server paradigm, significantly fewer are aware of how to connect applications together to support an event- driven or broadcast paradigm. Yet, for certain B2B scenarios, this type of integration can be one of the main challenges of enterprise application integration.

Fortunately, enterprise messaging offers an attractive solution to this problem. Distinct from a remote procedure call (RPC) approach, messaging has several important benefits that make it an attractive solution for enterprise-level integration. In this feature, I briefly review how messaging is different from traditional RPC-based approaches, give a brief overview of the messaging market and its players, and then give a more detailed example of how a company might use messaging.

Generally speaking, applications can talk to each other in two ways: synchronously and asynchronously (see Figure 1). Most readers are familiar with the former, such as RPC. Messaging-based integration, in contrast, is asynchronous. While not new, messaging is not as prevalent as something like RPC and so it may be something of a mysterious technology. Before we dive into its details, let us first review the basic aspects of synchronous and asynchronous communication.

Synchronous Communication

Communication that is synchronous consists of the traditional request-reply protocol found in client/server systems. A client makes a request of a server and that server responds with a reply. As a user, you encounter this every time you browse the Web. Your Web browser makes requests of Web sites, which serve Web pages. Similarly, applications that interface traditional databases process queries by communicating with interfaces and APIs provided by the DBMS.

In terms of enterprise application integration, RPC-style communication is the most common synchronous solution. By referring to RPC, I am including distributed object architectures - like CORBA and Microsoft's COM - that rely on RPC-style communication to enable objects to talk to each other. More recently, the platform-neutral Simple Object Access Protocol (SOAP) represents another form of synchronous communication. SOAP enables Web-based enterprise integration using a simple XML-based protocol for routing data between enterprises. (See Resources.)

Synchronous communication has three major drawbacks. One is the dependency on a reply from an information consumer; sometimes a reply is unnecessary - and can even get in the way - as I will describe shortly. The second drawback is that the client must initiate the communication. And in some push-like contexts, this responsibility simply does not make sense. The natural flow of information in these scenarios is for a broadcaster to push out updates to multiple listeners. Finally, a third stumbling block is the challenge of fusing together client/ server technologies that are different and costly to integrate.

This last point is a boldly underlined shortcoming of today's distributed objects solutions. A client application (running on behalf of your company, A) may exist in a CORBA distributed object system while the server side (running on behalf of company B) is a COM system. Bridges do exist, but what happens when the COM-based server needs to integrate with another partner that uses, say, Enterprise Java Beans (EJBs)? Do you buy another bridge? Or what happens if your partner is using proprietary technology? You already have a number of headaches caused by the lack of smooth, cheap integration between distributed object-based systems. In search of a solution, vendors have been migrating toward SOAP-based solutions as a way to achieve integration over the de facto portability assured by the Internet.

Don't get me wrong. Synchronized communication is essential in many contexts; applications querying databases and users querying the Web are just a few such instances, and there are many others as well. Consider server-side transaction processing of a bank transaction. Suppose a person uses an online bank application to transfer $1,000 from account A to account B. On the back end, the bank may use a distributed object system that processes this request by invoking business logic between distinct object servers. No practical way exists to process these kinds of transactions unless synchronous communication exists between the distributed objects. Any other way simply would not make sense. Can you imagine a user being told, "Oh, your request for a funds transfer is being processed - check back tomorrow"?

Asynchronous Communication

Still, you do have cases when asynchronous communication is not only desirable, but essential: It is a logical choice when servers need to push out data to your clients and when replies from those clients are not needed immediately. What's more, asynchronous communication is essential when the publisher of information cannot afford - for the sake of time or reliability - to wait for a reply. As mentioned before, stock market trading systems are a good example of where asynchronous communication makes sense. Market prices need to be pushed out to clients as they change, not the other way around.

But let's move away from the arbitrary example of the stock market and focus on the B2B aspect of asynchronous communication. When enterprises want to push data between internal systems or between partner systems, asynchronous communication often makes sense. The reply from clients is important, but its contents are not. For example, when pushing a group of new product prices out to the information systems of retail partners, the publisher of those prices is not expecting any reply to its price announcement; it just wants to make sure that partners have received the new information.

There is an indirect advantage to not requiring a reply. Suppose a client system has crashed or is heavily loaded: From the information producer perspective, you obviously want to be sure that your data reached the consumer, but you can be satisfied with the idea of it being a queue of data to be processed. If you had employed a synchronous form of communication, your server would have been stalled waiting for the overloaded (or unavailable) client to respond. This wait would hold up processing on the server side and - depending on the server-side system - might eventually lead to time-outs or even errors.

Asynchrony Through Messaging

Enterprise messaging solutions have emerged as the most popular form of asynchronous communication between heterogeneous applications. By heterogeneous, I mean applications that may be run by various enterprises, may be on different operating systems, or perhaps under different middleware systems. Messaging, then, acts as a distinct piece of middleware (also called MOM - message-oriented middleware) that connects these applications. Messaging is most attractive in contexts where heterogeneous applications need to periodically and automatically shovel data between each other, or when the nature of their integration is event-driven.

For example, in a recent research note, the Gartner Group has observed that B2B (and B2C) applications are getting more "streamlined" and that fine-grained workflow processing is becoming a common requirement. Workflow is a typical event-driven problem: When an event occurs, an approval or a notification is required. The natural form of communication here is to publish events to which various listeners subscribe. Understandably, messaging has been positioned as a solution for managing streamlined workflow.

System Architecture

In its most basic form, a messaging system consists of information producers, consumers, and messaging queues that act as intermediaries between producers and consumers. Information producers push data into a messaging queue. The messaging system, in turn, assures the data is eventually delivered to the clients. Thus, the queue acts as a "virtual transfer station" where produced information waits to be consumed. The queue part can actually be more complex than simply a means to connect a single producer and a single consumer. For example, multiple consumers can consume ("subscribe") to produced messages. Furthermore, the queue acts as a logical point where data can be reformatted or transformed into a format that the consumer understands.

Asynchronous communication in the messaging industry is referred to as a "loosely coupled" approach toward application integration. Unlike the tight coupling that occurs when distributed objects from disparate systems bind to each other and exchange data, loose coupling is the notion that connectivity between publisher and subscriber uses an intermediary - a third party - to broker communication between the two.

However, this loose coupling also exposes an oft-noted weakness of messaging: a lack of security. Despite assurances from messaging providers that secure solutions exist, messaging systems nevertheless represent a third party - a liability - in what may be secure, confidential communication between two systems. And if nothing else, that third party represents a risk because the data is momentarily out of control of both the data provider and the consumer. While it is possible to encrypt the data being sent, this complicates (and may even invalidate) automatic message transformation or formatting normally provided by the messaging system.

Implications of Messaging

The surge in messaging technologies has led to a number of related solutions. For example, message queuing and message brokering systems can provide a layer of robustness and scalability that guarantees that a message will get through, even if it is temporarily queued. Thus, a queue acts as a buffer between applications that are not always available.

Another interesting side effect of the message broker phenomenon is that it has eliminated the need to integrate applications on a point-to-point basis. Under synchronous-based solutions, applications are integrated by connecting APIs together. But doing this for every integration scenario implies a lot of integration fibers between the two applications and makes the evolution of those applications tricky, if not impossible. This intricate process is like connecting every peripheral card not only directly to each other, but also to the memory and the CPU! The message broker has thus emerged as an "information bus," a one-stop shop for distributing data between disparate applications.

Yet a third interesting advantage of messaging is that its intermediary role makes it a natural point where data can be translated from producer to consumer. For years companies have used electronic data interchange (EDI) approaches to integrate data where reformatting is necessary. While EDI has retained its popularity as the Internet has evolved, it lacks good support for security (less powerful than Secure Sockets Layer-based Internet transactions), it is costly to maintain for both client and server, and its cryptic formats make debugging difficult.

Messaging systems, on the other hand, provide automation that reformats data as it passes from source to destination. For example, source data that needs to be padded with blanks so that character strings are a fixed length have this need for reformatting. On messaging systems, formatting options abound - for example, products such as IBM MQSeries enable integrators to construct flexible formatting rules.

The Market and Its Players

The market for MOM is made up of a number of vendors, including IBM, Mercator Software Inc., Microsoft, New Era of Networks Inc. (recently acquired by Sybase), Progress Software Corp., Talarian Corp., and Tibco Software Inc. Most analysts agree that IBM is the Goliath of messaging, controlling more than 70 percent of the current market share. The others have their niche, but the IBM MQSeries product family cannot be ignored, and industry players related to the messaging market are nearly always are forced to reckon with it. Still, recent Java technology introduced by Sun Microsystems may shake up the market to some degree, as I describe a bit later.

Interestingly, some integration between market players does exist. For example, Tibco offers "adapters" - including ones for the MQSeries - as a means to achieve enterprise integration. Also, IBM works with Mercator to route messages between publishers and subscribers through Mercator software.

The Java Factor

The Java Message Service (JMS) API does for messaging vendors what the JDBC API does for database vendors; it provides a standard interface for incorporating messaging into Java applications. As Figure 2 shows, the JMS acts as a wrapper around messaging vendor technology, allowing developers to focus on coding - not the particulars of one vendors' API. In addition to basic messaging functionality, the JMS works with other Java technologies, such as the Java Transactions API, to provide features like distributed transaction support. However, JMS does not provide other features, like load balancing, fault tolerance, security, or quality of service guarantees.

JMS supports two general messaging scenarios, point-to-point and publish-and-subscribe. Point-to-point is generally used when messaging occurs between a single producer and a single consumer. Under this scheme, a synchronous form of messaging is actually possible. With publish-and-subscribe, it is possible to have a single producer associated with multiple consumers and to enable a multicast style of communication.

Standards like JMS force the industry to ensure that it supports the fundamental features of a messaging service. JMS has been implemented by a number of vendors, including BEA Systems Inc., IBM, Oracle, and Progress. Other companies, including Novell Inc., Tibco, and Sybase, have also endorsed JMS, and some open-source efforts are underway. The emphasis on JMS in this article is not meant to imply that it is the only solution out there. However, it cannot be ignored and, given that many other vendors (including market-leader IBM) are involved in shaping it, this technology is important to underscore.

For an example of a typical messaging scenario, see the sidebar A Messaging Solution."

And the Message Is ...

The intention here is not to say that messaging will solve all enterprise integration issues. It is unquestionably useful and even essential in some contexts. And for many people used to RPC-style distributed system integration, it represents an alternative. Most likely, as enterprise systems continue to integrate, a healthy mix of both synchronous and asynchronous modes of communication will exist.

However, as systems become increasingly integrated, at finer and finer levels, the risk of system unavailability looms even greater. For any company, being at the mercy of a downed system you've never heard of is especially alarming. The advantage of asynchronous messaging here is undeniable. What may eventually evolve is a landscape where enterprises rely on messaging as a primary integration technology and only fall back on synchronous-style integration when they really need it.

Return to Article