Agents Are Watching

E-business process monitoring - as enabled by intelligent agents - may soon eclipse enterprise system management as the guarantor of business process continuity

By Elmar Husmann, Thomas Schmitt & Thomas Schuler

In his book The Society of Mind(Simon & Schuster, 1988), AI pioneer Marvin Minsky proposes that “mind” is a collection of many smaller processes called agents. On its own, an agent can do only very simple things. But when it joins other agents to form “societies,” intelligence and complex behavior emerge.

We can also apply this concept to e-business. What makes e-business so special is connectivity and networking; the fact that by simply connecting to the Web, we can interact with the whole Internet community. This interaction is limited neither to one-to-one transactions nor to the pure exchange of information, those things that make the Internet the first virtual global market.

As in Minsky’s “society of mind,” in e-business, complex processes may develop from the interaction of simple components — for example, when suppliers form digital marketplaces. Similarly, companies that once regarded the Web as just another sales channel now interact with their customers, multiple consumers aggregate their purchasing power at sites such as letsbuyit.com, and customers use software shopping agents to compare prices and delivery conditions.

In an article in the March 2000 Wired (“Capitalist Econstruction”), Chip Bayers predicts a new era of “bot commerce” in which prices are negotiated dynamically, and customers rely on shopping agents rather than reseller brand names. If Bayers’ prediction comes to pass, service quality and ease of use will soon matter more than established brand names: When the next offer is just one click away, the stability, security, and functionality of customer-facing portals as well as back-office business processes become even more crucial. For example, in a March survey of online consumers, the Boston Consulting Group found that convenience was the most important motivation for initial online purchases. However, the respondents also cited performance, stability, and security concerns as important drawbacks, with many reporting severe problems such as the wrong products arriving, unauthorized credit card charges, unacceptable delivery times, and system crashes.

Controlling E-Business

Legacy IT developed over the years into a complex landscape characterized by interconnected systems (most of them custom developed), nonstandard interfaces, and time-based offline synchronization. This world is one of batch processes and fixed EDI connections, with most of the data and information flow uncoupled from direct business process execution. Structural complexity was high but the need for flexibility was low.

Now, state of the art IT, based on enterprise resource planning (ERP), data warehousing, and customer relationship management (CRM), is less complex from a technology perspective but has much more inherent business complexity, which offers the opportunity to shape applications to business needs. Furthermore, business-process automation and integration have increased significantly.

Now, e-business is adding even more elements into the mix, such as content-management systems, e-commerce platforms, and collaboration engines. In some cases, these architectures can include as many as 10 or more logical components without taking external and back-office systems into account.

This change in IT parallels the fundamental transitions in business models. Forrester Research recently proclaimed the “death of IT” when fast-changing business processes span multiple companies. According to Forrester, this development may drive IT management “outside” into an external technology environment, giving rise to a new market of service providers and leading to a replacement of internal IT management with operational business-process ownership by business departments.

Enterprise systems management (ESM) improves traditional systems management by broadening the view from single, specific systems to the enterprise’s complete installed infrastructure. Thus, ESM tools such as Tivoli Systems’ Tivoli, HP’s Openview, and BMC Inc.’s Patrol ostensibly provide control over the complex, changing IT landscapes we’ve described. However, according to Gartner Group, 70 percent of ESM implementations did not meet expected goals within three years. And Hurwitz Group says the percentage of successful ESM projects “closely approximates zero.”

Because of poor IT service process integration and project methodologies, ESM does not provide a business- oriented view of the full enterprise — which is an absolute requirement in an e-business environment. This fact is obvious in environments containing large applications with significant inherent business complexity, such as ERP systems. Thus, ESM is useful strictly as a tool for technology-oriented IT service providers.

Based on these trends, Forrester and Gartner Group are both calling for a new kind of technology management at the process level: e-business process monitoring. The justification for this call is that in e-business environments, automatic interpretation and routing of online information about the status of business process execution will become routine, and messages automatically deriving from inside the company will seamlessly integrate with those from business partners.

Established online brands are already starting to integrate external services. Take the example of Sony Corp., which recently announced a new credit card, mileage program, and online bank. For the online Sony customer, the lines separating the different business divisions of Sony and its partnering bank are no longer visible. Thus, IT service management for such complex services has to guarantee the continuity of business processes regardless of the business unit or service partner performing the action. Only better communication and information integration can achieve this goal.

Event Handling

ESM is not a total loss, however. The general concept of event handling as applied in ESM — detecting events in decentralized systems and routing them to central information points — is still viable. For example, an organization could theoretically use event handling to record e-transaction histories and search databases for errors and inconsistencies.

These goals imply the gathering of real- time information about the status of diverse business objects from multiple data sources, a difficult task when using classical “pull” techniques such as data warehousing. When the landscape of data sources changes constantly — or sources are not precisely definable — no extract-and-load procedure can help. In this environment, extracting meaningful business information from raw business or technical data is a very inflexible and complex procedure. For e- transactions, it may even be impossible.

Event handling, however, is based on an opposing concept: waiting for information to be “pushed up” in a standardized manner. In this approach, you define events in an object-oriented way and convert application-specific data into standardized messages. By grouping these objects into a hierarchy of classes, you can define event-triggering rules on several hierarchical levels with full inheritance properties. This approach reduces complexity and provides much more flexibility.

Classical ESM is designed to address availability information, but you could think of other types of applicable information as well, such as processing confirmations for online orders or information about content consistency between the catalogs of an online retailer and its suppliers. Regardless of the increasing number of e-business applications, however, the basic types of e-business collaboration are limited. Our aim is to define what type of information is needed to monitor and control an e-business and to do so independently of applications. ESM already provides this function for technical information to help manage complex, heterogeneous IT landscapes. Why not do the same for business information to address heterogeneous business application and service provider landscapes?

A prerequisite of this goal is to decouple the logical layers of infrastructure and raw information from the modeling of business rules and information distribution. Using the same information-exchange standards, business partners can more easily integrate the monitoring of their interacting applications. The goal is to create an information- processing environment that is largely independent of data sources and individual data formats.

What We Need to Fix

Event handling looks like a promising solution for e-business management and monitoring. But current event-handling ESM solutions still fall short of the goal for several reasons.

Events lack relevant business content. The first problem is that in these systems, events lack relevant business content in terms of business object information or business partner identification. Information about system availability — the key domain of ESM — contains technical parameters such as server workload or database activity, which are not suitable for reconstructing business pro- cesses. Thus, ESM vendors will have to rethink their event classes and the way they access the information. It may be desirable to define Internet standards for operational business information as well as for the business object itself.

Events relate to physical elements and applications, not to logical elements. E-business monitoring implies the replacement of structural complexity with business complexity; thus, business-process steps will replace physical elements, such as servers or applications, as structuring elements. (Such logical elements could include “online orders income” or “credit card checking.”) When an event is linked to a logical, and not a physical, element, event generation should likewise be clustered by logical (not physical) elements — which ESM systems don’t do. Event agents should be responsible for a specific information type, not an application as they are in IT-focused ESM.

Presence of event “storms.” Business-oriented monitoring depends on having the right information. But in existing ESM applications, the production of events can reach critical dimensions, making it hard to distinguish relevant information from the “noise.” How could you distinguish, for example, the mission-critical system failure that affects your whole customer service desk from the sum of events on downtime and errors in your back-office processing environment? In actual ESM installations, this task is usually a manual one for a human operator, supported by a few rules in the monitoring console.

Business events will contain very condensed information, so the system has to dig for the right information at an early stage. Searching a large number of unspecific events on a higher level for relevant information, as ESM systems do, won’t work.

Events have only a single resolution level. In traditional ESM, an event has a single resolution level. It is possible to condense several events into a new one for event filtering purposes, but this process is irreversible. Finding the event correlation path in which business information was created from technical events is crucial for detecting underlying problems.

Events are handled in a pyramid structure. The architecture of ESM tools is designed to collect events from a complex IT landscape. In this architecture, a local agent controls a group of infrastructure elements — such as servers or network nodes. This agent passes events via gateways to a central event-managing server. This approach is appropriate when dealing with a large number of nonintelligent devices in a fixed structure, with each device treated as a homogeneous black box.

Event management servers are structured in a pyramidal fashion and contain sets of rules to propagate events and start actions. This pyramid decomposition is useful for decomposing events by type, geographical location, or organizational property. However, it does not enable flexible event management based on logical business areas. In this approach, event servers simply route events and perform simple correlations, not apply complex business rules. Instead, event processors should be linked in a true network to allow distributed event handling according to business and logical areas.

Elements of E-Monitoring

We can overcome these problems, however, by constructing a true business- oriented e-monitoring environment. (PricewaterhouseCoopers is currently evaluating such an environment in prototype.) This environment has these characteristics:

Agents for detecting business-relevant events. In this architecture, on the lowest level, intelligent agents create events. These agents provide a range of standardized information services, such as testing performance or data consistency in regular intervals, monitoring process chains, checking the status of business objects, and interpreting error messages. Agents might also perform purely business-oriented checks, such as finding delayed orders or open invoices.

We classify these agents into groups according to their purpose. They contain sets of generic rules about how to monitor a logical process element, such as checking product catalog consistency in multiple supplier systems.

Agents address their requests generically to application connectors placed on top of, or as part of, business applications such as SAP or Siebel. The agent can activate the connector itself in order to hand over application internal events. These connectors are based on direct-access APIs, such as SAP’s application link enabling (ALE) interface.

These intelligent agents gather only very precise and already condensed information. This action is triggered by rules or structural information passed over from a business layer through inheritance. Once configured, the agent watches for the requested type of information and pushes it up to the event-handling network. Pricewaterhouse- Coopers applied this function in a test environment to an e-commerce storefront and its related back- office order processing. The agent watched for transaction performance, availability of key components, and automated processing steps, as well as time constraints on the further processing of open orders and similar business information.

Distributed event-handling networks. Agents provide their full benefits only when operating in networks. If you consider an agent to be a virtual expert on a certain subject, then you can achieve a full, enterprisewide overview only by linking it to an “expert network” containing translators and other subject experts on several layers.

This network includes a series of intelligent event-processing “cells,” which you can set up and maintain through a Web interface. (In the test environment, PricewaterhouseCoopers used the Master- Cell technology from a company called IT Masters.) Each cell represents a business-process element or logical group with an individual rule set; in our online shop example, these elements included the shopping front end, order entry, supplier connection, assembly, and shipment. Depending on the rules, a cell can further process events, generate new ones, start actions, generate messages, and so on. The presence of several cell layers can lead to complex problem decomposition, and many of them can be nonspecific to individual infrastructure. The shopping front end in the test example was further decomposed into product catalog, order and shopping basket creation, customer account maintenance, and availability checking.

Furthermore, several cells can translate and condense technical events into business events. An important feature of the cells is that they maintain a history of event creation; the full information is not attached to the condensed event but temporarily stored in the cell. This approach enables drill-down from a top-level business element to the detailed information at the beginning of the chain. For example, a top-level business event that indicates a serious failure in the online front end could be traced back to the order creation step and from there, down to its technical roots. Questioning the cells in real time invokes this process. That means that in the top-level cell, only the condensed information about the serious failure in the shopping front end is processed. A root-cause analysis can be undertaken by reconstructing, with the help of all involved cells, how this event was generated. Thus, the information on top is highly condensed and purely business oriented. The detailed root-cause information is distributed across the net of cells.

Rule inheritance is another important feature; changes in any level are passed along to their underlying sublevels. This feature also enables the configuration of agents to provide increasingly specific information.

Working With Business Events

The cell network can have multiple hooks in order to seamlessly integrate with messaging, trouble-ticketing, and decision-support systems. Reporting failures and problems with more business content enables sophisticated automated routing of messages and escalation procedures. Furthermore, this information exchange can integrate with those of business partners through business gateways. The online store in our example could automatically signal its unavailability to the provider of the Web business in which it is embedded.

Meaningful business information can further empower end users and customers, providing them with personalized online status reports on IT systems, business processes, or individual transactions. Customers might automatically be informed when delivery is delayed for a week and support people alerted when a client encounters serious technical problems.

Online reporting is also a very important application for recording e-transaction history and process status. To provide this service, business events are stored in a central data repository through which managers can use business intelligence tools to detect inconsistencies, find key performance indicators, or do trending analysis. This capability offers further possibilities for providing end-to-end e-business monitoring, such as correlating business information from sources such as data warehouses. For example, using the e-business management system, an online retailer could analyze clickstream behavior, information about performance and user load, delivery times, and supplier reliability, all through a single Web portal. In a management-reporting context, this information is useful not only for proactive IT management and monitoring service-level agreements with outsourcing partners, but also for enabling a consistent business-monitoring and value-based management approach.

Know Your Stuff

IT management needs in the e-business era will give rise to very sophisticated ways to automatically exchange business-relevant event information among companies, track transactions, dig for inconsistencies, and seamlessly integrate monitoring and messaging environments. Transparency is a crucial issue: In e-business, you’d better understand what you are doing.

Rate This Article Rating: 1 (best) 2 3 4 5 (worst) Comments: Optional e-mail address:

Elmar Husmann (elmar.husmann@de.pwcglobal.com) is a consultant in the Hamburg, Germany, office of PricewaterhouseCoopers focusing on IT strategy, management, and organizational issues. Currently, he is doing IT organizational design for a large e-banking start-up.

Thomas Schmitt is a principal in the Zurich office of PricewaterhouseCoopers. He has more than 10 years of experience in IT consulting, service management, and in project management for large-scale ESM implementations.

Thomas Schuler is also a consultant in the Zurich office of PricewaterhouseCoopers; he has 10 years of experience in all aspects of IT monitoring with a focus on large ESM implementations.

RESOURCES ITMasters: www.itmasters.com