Matching Patterns

Patterns in historical data are the lifeblood of business intelligence and knowledge

By Girish Keshav Palshikar

UPS AND DOWNS

Patterns Reveal Connections Users in diverse domains often describe interesting patterns that they expect to see in the databases collected by their systems over time. They often use high-level domain concepts and various temporal and logical connectives to describe these patterns. Some examples of temporal patterns in end users' language include: Whenever the price of a stock is very high or very low, the stock is trading rather low. Large purchases of a stock within a short time interval lead to a sharp rise in the price. As the market price of a stock increases, so does the volume traded. The selling tendency increases whenever a stock price goes above the previous maximum. A failing bank has a large number of large nonperforming loans that eventually become loan losses. Large increases in total loans within a short time, along with only small changes in provision for loan losses, are followed by a period of high-net loan losses. Whenever the furnace temperature rises rapidly, the yield quality somewhat deteriorates.

A critical analysis and deep understanding of "historical databases" form a backbone of many special business-critical tasks: decision making, surveillance and monitoring, performance evaluation, risk analysis, fraud detection, resource management, diagnosis, planning, and forecasting. Automated business systems collect vast amounts of data over long periods of time in stock markets, banks, sales and finance departments, insurance companies, manufacturing industries, and so forth. The long-term data - perhaps from several years past - typically resides in a data warehouse and short-term data - the current year or immediate project - may be available in online databases.

Making significant business decisions requires extensive knowledge about your business and your collected data, which is often expressed in terms of patterns in the data and its relationship with business phenomena, activities, and decisions. After all, as Hegel said, "Those who do not understand history are condemned to repeat it." To say that detection, measurement, understanding, and effective use of patterns in historical data form a core of business intelligence and knowledge would not be an exaggeration. More fundamentally, the ability to detect, measure, and analyze patterns is a crucial human ability and the basis of perception, understanding, and learning.

For example, a manager may offer incentives such as discounts when stock levels gradually increase while demand rapidly dwindles. Or as another example, a surveillance manager in a stock exchange normally expects the trading in a particular security to be rather low during a period when its price is high, or very low when the price is relatively steady. The manager may be interested in a system to define such a pattern (at this level of abstraction, away from the database details) and that will generate an early warning about the exceptions to this normal trading pattern. Then the manager may use further data analysis to either confirm or eliminate the possibility that the detected exception to the pattern constitutes some suspicious activity.

Users need tools to express and detect patterns in given databases, particularly because the databases are often quite enormous and manual inspections of data are well-nigh impossible. Management information and decision-support systems in such applications often provide a range of services, typically in the form of canned queries, reports, visualizations, and special-purpose computational and analysis facilities (for example, based on time-series) so that the decision makers can make informed and intelligent decisions consistent with past observations.

The often unstated assumption is that these tools will only provide support to the IT experts to detect and understand the hidden patterns. Unfortunately, many of these tools do not really have a clear underlying notion of what patterns are or any special facilities for detecting and measuring them. But the primary responsibility lies with the database users and their skills.

In this article, I explore the notion of a pattern, particularly a temporal pattern, and suggest an approach for matching a known pattern against given databases. This approach derives from the research of my organization, Tata Research Development and Design Centre (TRDDC) and its practical applications. TRDDC is the research and development division of Tata Consultancy Services, a software company in India.