Intelligent Enterprise

Better Insight for Business Decisions

RSS
Webcasts
Whitepapers
Subscribe
Home

June 28, 2002

Expect the Unexpected

In a world full of uncertainty, skillful forecasting can minimize disruption due to unanticipated events

by Seth Grimes

Aristotle wrote 2,400 years ago, "It is likely that the unlikely will happen." This familiar notion was a touchstone of the Federal Forecasters Conference, held in Washington, D.C. in April 2002.

The annual conference covers predictive modeling applied to diverse sectors, including energy demand and markets, trade, transportation, agriculture, healthcare, and others. This year's focus on discontinuity and uncertainty in forecasting was understandably motivated by disruptions due to terrorism, but many, many other sources of disruption concern government planners. Speakers repeatedly cited Sept. 11th's effect on sectors such as aviation and energy markets, but they also addressed more mundane subjects that may have a much greater long-term effect on our lives, such as global climate change, population aging, and changes to federal policies and regulations.

The federal forecasting community, working with academia, is developing techniques to extend model-based forecasting and time-series techniques to accommodate disruptions. We can all learn from their progress. I'll cover highlights of the conference sessions I attended, leavened with my findings in additional research I've done on the topic. First, I'll note that forecasters are refining their use of techniques from related areas of unconventional mathematics developed decades ago. These methods include fractals for modeling systems characterized by jagged patterns that are repeated on a variety of scales and catastrophe theory for modeling the effects of truly radical change; for example, the world-altering effects of a comet that hit the earth in the Cretaceous era, possibly leading to global cooling that killed the dinosaurs. I'll cover these latter topics in a future column.

Framing the Problem

One of four Federal Forecasters keynote speakers, Ed Spar of the Council of Professional Associations on Federal Statistics, made a distinction between forecasting and projection that neatly frames the disruption problem: Many people simply extrapolate from past behavior into the future. Such projections are rendered worthless and unsalvageable by a change to basic circumstances. True forecasting, whether based on time series or regression analysis, is much more robust.

Another keynote speaker, George Washington University economics professor Herman Stekler, reviewed forecasting problems and forecaster failings. Economists have failed to predict recessions, meteorologists haven't predicted storms, the military hasn't foreseen the outbreak of war, and technologists once predicted that there would never be need for more than half a dozen computers or more than 640K of memory in a PC.

Stekler said, "Formal models can't capture unusual events" — whether they are foreseeable or unforeseeable, — so "we have to rely on analysts," but models and analyses are vulnerable to misinterpretation of data and forecaster bias and preferences. First, statistical interpretations are usually Bayesian, deriving predictions from prior occurrences, which is clearly a problem where a conceivable event has never before occurred. Forecaster bias "leads us to select evidence that's consistent with preconceived views." And Stekler explains the preference problem with the observation that "no one likes to bring bad news to the boss."

Stekler offered a number of simple solutions to remedy forecasting failures, in particular, enhancing modeling to accommodate uncertainty, which entails accepting the cost of false positives (predictions that don't come to pass) and working to identify bias. To predict the outcome of structural change due to, for example, changes to government regulations or policies, Stekler advocates use of dynamic, interactive simulations — "war gaming" — that builds on, but surpasses, the accuracy of judgmental forecasting by panels of experts and scenario and outcome analyses.

Accommodating Disruption

Scenario analysis simulates the real world using systems of equations. A set of particular values taken by the input variables constitutes a scenario, which the model transforms into an outcome expressed as a set of indicators. (I covered this type of modeling in my Feb. 16, 2001 column, "Simultaneous Equation Models.") Staff from the Department of Agriculture's Economic Research Service presented some of their scenario analysis work in a conference session I attended. Researcher Ralph Seeley, for one, uses the General Algebraic Modeling System (also known as GAMS) to simulate 24 commodity markets in 44 regions. Seeley has found that handling new scenarios that result from structural changes "may require increased capacities for the models, analysts, algebraic techniques, and overall system." One accommodation Seeley makes is to transform new scenarios into known situations.