Breaking the Cycle of Failure

A business process model approach toward requirements analysis may improve your data warehouse project's odds of success

By Mark Riggle

Software projects fail at an alarming rate, and the statistics for data warehouse development projects aren't any better. In fact, one IT executive was asked, "Why has your company built eight data warehouses? That seems like a lot." The executive's reply? "We had seven failures." Such failures are often the result of a breakdown at the requirements analysis level.

If you inadvertently build the wrong system, then you must make time-consuming and expensive corrections to it. Your project team must keep repeating development cycles until it finally determines the real requirements, resulting in missed deadlines and higher budgets. Many of the top data warehousing mistakes that people list are caused by failures in the requirements phase.

In a sense, this situation is surprising for a data warehouse project because it seems that it has a predefined universe of business users and sourcing data. Business users know how the enterprise operates and developers are familiar with or have access to the automation systems running that business. In turn, the data warehouse should extract the data from the automation systems in a form that helps users understand the business. What could go wrong in the creation of a data warehouse and connected business intelligence (BI) systems in terms of requirements analysis? Plenty! I will explore this failure and detail how to solve it using a methodology for specifying data warehouse requirements based on business process models.

ORIGINS OF FAILURE

The most common and expensive-to-repair failures in software projects originate in the requirements phase. Postmortem studies of these systems cite inadequate, inconsistent, or changing requirements throughout the project life cycle as the most important reasons for these failures. Those studies use a rather broad definition of failure: extended schedules, running over budget, and failure to meet stakeholder needs. However, extended schedules and higher budgets usually reflect the reworking of large parts of the project to meet changing requirements. Failures in data warehousing projects also fall into this difficulty. Incorrect requirements create wasted work and, consequently, higher costs.

Why are the requirements in a data warehouse project so difficult to determine? Two modes contribute to the requirements failure:

1. User centered. In this failure mode, some requirements from your users are inherently changeable and unknowable a priori. Your users don't know what they really want and can only recognize it by seeing what the data warehouse can produce and then responding that they want something else. The users cannot foresee how and why they will use the information, thus they need several iterations to explore that solution space.

2. Developer centered. For this failure mode, misunderstanding between your business stakeholders and developers is the root of the problem. This situation usually occurs because the developers are not well grounded in the business, and the stakeholders do not understand the development process. The critical issue is that the developers are not familiar with the stakeholders' basic business assumptions and structure, which leads to very different interpretations of the business user's needs. Thus, the developers need several iterations to explore the problem space. Without an understanding of the correct problem space, the right solution is impossible.

In many software and data warehouse projects, people make the mistake of attributing communication errors to errors of the "unknowable." The data warehouse development staff blames the stakeholders, and the stakeholders blame the developers. From the developers' point of view, their misunderstanding of business users is often attributed to the users changing their minds, or the user-centered failure mode. However, this latter situation is actually good because you can always improve your communication problem.

To resolve your developer-centered failures - the communication problem - you need to change the nature of the communication between the parties. Although many techniques center around joint application development to improve communication or require close consultation with subject matter experts, continuing high failure rates certainly indicate that these techniques aren't sufficient. Very often you may use data models to facilitate communication, but they are often hard for the businesspeople to understand. Instead, the development staff determines data warehouse needs by asking the users for their needed queries and for existing useful reports. Clearly, the requirements communication needs additional help.

SOLUTION DIRECTIONS

The requirements problem is deep-rooted and must be looked at in depth to find out what exactly defines a business. Michael Hammer, a business process reengineering advocate, defines a business as a set of processes that should add value to the customer and make a profit for the business. Hammer refers to business processes as opposed to the processes used in the software information systems (see Resources). If a business is a set of business processes, then all automated systems, including the BI, data warehouse, and OLTP systems should reflect that process structure. At the heart of the communication problem is the fact that business users understand their business at the business process level while data warehouse developers mostly understand it in terms of data structures. Obviously, a solution must address this dual view.

What is a business process exactly? A business process is fundamentally a time-sequenced list of activities executed by an agent that consumes resources and produces output. In addition, a hierarchical relationship of processes exists: You can compose a process from many subprocesses and this abstraction of subprocess hierarchies is crucial to your requirements' understanding. You can give a business process a formal model and diagram. Figure 1 shows a process of assembly and test and Figure 2 shows that process in its hierarchy.