Intelligent Enterprise

Better Insight for Business Decisions

RSS
Webcasts
Digital Library
Subscribe
Home

November 15, 2002

The Paradox of ROI

Do ROI studies really help organizations make better buying decisions?

by Joshua Greenbaum

As the recession began to take hold in 2001, the notion of return on investment (ROI) gained increasing interest. As the effects of the recession continue, ROI studies have become as common as a product brief or an FAQ. The rationale is obvious: No one can afford to buy technology for technology's sake anymore. If a product can't survive an ROI analysis without a "measurable" return, then companies shouldn't buy it.

The advice is sound, and the reasoning is good. There's just one problem: Most ROI studies have serious flaws that, at best, make it hard to translate the study's ROI numbers to your organization's real-world case. At worst, many of these studies are poorly designed and executed and serve to mislead the uninformed. That doesn't mean vendors shouldn't produce ROI studies or that customers shouldn't use them to make buying decisions. They're still extremely useful. To paraphrase Sir Winston Churchill, ROI studies are the worst possible way to make a buying decision, unless you consider the alternatives.

I must have done a dozen such studies since the ROI boom began, and even in the best of all possible worlds, it's been hard to develop an ROI study that's truly meaningful to every prospective customer. Impossible? No. But as the consumer, you should be aware of what to expect from vendors (and consultants) that offer up these studies as the foundation for your purchasing decisions.

The Problems With ROI

Let's start with the fundamental goal of all ROI studies: to describe how existing customers derive value from the use of a particular software product, and to extrapolate a generalizable model from those experiences that can predict the degree to which a new user will derive a measurable ROI. The problems start right away. In order to do an ROI study correctly, you need enough customers with enough experience who are also willing to talk at length. And these customers must have carefully measured the before, during, and after status of their businesses and IT systems in order to accurately describe their ROI experiences. Good luck.

What constitutes enough customers is the first problem. For those who were asleep during their statistics classes in college, sample size is one of the keys to the validity of a survey. The paradox of sample size is that you can model a tremendous amount of opinion with a relatively small sample: If you survey 1,500 adults about a given subject — for example, is the President doing a good job or not — you're well on your way to modeling how 250 million people will answer the same question.

But behavior is different: Complex behavior patterns, like IT software acquisition and use, are so different from one organization to another that asking questions about how software is used generates impossibly large sets of variables that are hard to measure, harder to quantify, and often impossible to analyze. Add to that effort the task of building that "generalizable" model and you've hit the first wall in the ROI dilemma: Few if any ROI analysts actually talk to enough customers to build that one-size-fits all case, which means that using the ROI study to predict your company's experience just became harder, if not impossible.

The validity of the sample is the next problem. I did an ROI study for a vendor that surveyed every one of its customers. The only problem was that the client had relatively few customers who were doing vastly different things with the software. The good news was that all but one customer could demonstrate some ROI from the product. The bad news was that the study ended up being a set of small ROI studies, each with a sample size of one. Good enough, it turns out, to predict some ROI for virtually every customer. But there was no single model, and with such a small sample size, predictability was hard to come by.