Intelligent Enterprise

Whenever the subject of business requirements for data warehousing and BI comes up, I try to bite my tongue because it's always at a time in the project when expectations are high and people are hopeful. I hate to rain on their parade, but this is one of those areas where best practices are often worst practices.

The idea that you can go "do" requirements gathering is canonical, but it's surprising and ironic how few practitioners actually believe in its value. This isn't a bias you want to expose to your clients, though. I find it particularly vexing that training sessions and conferences on data warehousing and BI usually have requirements gathering classes, taught by people who really ought to know better. I guess that's a subject for another day — why industry "experts" are content to disgorge training, for a fee, that they know is misleading, but is widely accepted.

Here's what I think is wrong with the whole notion of requirements gathering:

1. Separated by a common language: Those who "take" requirements and those from whom they gather them may use the same words, but they frequently mean different things. This is worse than not understanding someone. To assume you understand when you don't leads to mistakes. And this works both ways. They may completely misunderstand what you're proposing for them and give you specs for a system you're not building. A two-hour group kick-off and a one-hour interview are not sufficient.

a. This is really part of a larger problem of IT not understanding what people do. The beginning of a major project is not the time to get people speaking the same language, it should be an ongoing process.

b. "Ride with a cop" type programs, where IT people work with the functional areas they support, have shown real promise.

2. Observer effect: In the same way measuring a subatomic particle changes it, discussing requirements with the domain expert tends to shift their perspective too. The mere act of asking how this new system could help them starts a process of rethinking their work that generally happens AFTER you've interviewed them.

3. Kick the dog: People's perception of what is important is overwhelmingly colored by their most recent experiences. If you were the Secretary of the Treasury today, you would think that the most important thing is the mortgage crisis, even though we all know that a year or two from now, it will be about 23rd on the list of concerns. If the organization just made a large acquisition, you will be told that integration is key, but by December, it might be fuel costs. It is difficult to disabuse people of their priorities, even when you know they are just transient.

4. Fake requirements: "If you can just reproduce these 207 reports exactly, we will know that the system is accurate." In other words, I don't know what you're doing and I can't be bothered. This is a diversionary tactic meant to avoid giving time or consideration to the initiative. Unfortunately, project sponsors, often not knowing better, are quick to attach this requirement in order to see the project have some firm "deliverables." It's a little like the lost driver who refuses to consult a map and claiming, "I'm not sure where we are, but we're making great time."

5. The Theory of Limited Good: Participants who are weary from repeated and low-functioning initiatives in the past who appear to be cooperative but wait for just the right moment to raise an issue that wasn't covered, stalling the project. They have no faith in what you're doing and stand to lose prestige if you succeed.

I'm not suggesting you start a project with no requirements. Rather, use experienced people who already have a good handle on what organizations like yours have done successfully. Don't send a DBA out to get requirements; use an experienced interviewer who can detect and overcome the five problems described here. And finally, beware of those classes that pick on the "business people" in the requirements process as a cover for the methodologies that really don't deal with it very well. There is nothing wrong with business people; they are just busy.