Fear and Loathing in Project Management

How to insulate your data warehouse project team from potentially devastating business decisions

By Michael L. Gonzales

Executive Summary

Michael L. Gonzales As a project leader, you may be faced with politically naive behavior from executives, the IT department, or users. How can you implement a complex IT project involving disparate data sources and processing requirements in such an environment? The Dysfunction, Impact, and Feasibility (DIF) matrix approach can help you define interrelated, overlapping requirements, prioritize iterations, and provide nonbiased decision-making.

When you have a large IT project to implement, everyone - your company's executives, IT personnel, and users - wants a successful project. But, it isn't unusual for politically naive behavior from these same individuals to compromise the success of a project and leave reasonable professionals caught in the crossfire. However, you can manage these difficult situations by accurately identifying and prioritizing your project iterations.

I've experienced my share of volatile situations. In one account, I vigorously tried to persuade the executive IT sponsor that building a data warehouse without involving end users wasn't the best approach. My argument was that the project team should build the data warehouse using business requirements based on user community feedback, not IT assumptions. The executive sponsor's response was,"It may not be the best way, but don't we have the right to be wrong?"

In another account, I was leading a politically challenging project for a company whose executives didn't trust the new IT director and often overwhelmed him with their aggressive actions and attitudes.

In both cases, it was important to remove - or at least insulate - the project team and myself from the potentially devastating behavior of misguided or inexperienced executives. To that end, I have implemented a modified version of the Dysfunction, Impact, and Feasibility (DIF) approach, providing a statistically valid method for quantifying and prioritizing the project's iterations. Although the technique's origins are in process redesign efforts, data warehouse projects exhibit similar characteristics with numerous potential starting points and iteration opportunities.

It is best to use the DIF approach when you need to recommend where to begin a project and what the order of subsequent iterations will be. This approach lets decision makers set the project direction based on empirical evidence rather than emotion or politics.

The Right Approach

You must first determine if your project has multiple starting points and iterations. For instance, in data warehousing you may have many projects queued and waiting to start. User communities often besiege warehouse teams with competing requests. Marketing might believe its project is more important than the sales project, and sales might argue for priority over manufacturing. Process redesign efforts are subject to rival interests as well. These types of projects will benefit from the DIF approach. However, single-iteration projects are less suitable, such as writing a report or adding a feature to an existing application.

I most recently applied the DIF approach in a process redesign effort. The goal of the project was to simplify the complexity of moving data from disparate applications within the organization. This client had been experiencing problems with its data infrastructure and exhibited symptoms such as:

• Difficult ad hoc reporting because of disparate data
• Increased system complexity caused by disparate data and technologies
• Timeconsuming manual rekeying of data between systems
• Lack of integrated and accurate data sources
• A heavy dependence on IT for any report requirements.

If you attempt to implement a point-to-point integration strategy as a means to address the disparity, you merely exacerbate the problem, resulting in classic interapplication spaghetti code, of which this client was a prime example.

No matter how good the programming team, the combined growth of integration points creates an unsustainable architecture. Point-to-point integration is simply not scalable and often easily broken with any change requirements. And, to make matters worse, point-to-point integration code often develops in a vacuum specific to the task at hand and is made with inconsistent tools.

As shown in Figure 1, the environment encompassed numerous applications, major external process feeds, and several "data warehouses," as IT sympathetically describes them. The real architecture is far worse, more complex and convoluted than I can portray here. Table 1 shows a hint of the severity.

Two formerly independent companies are now attempting to become one synchronized, competitive leviathan. Aside from the sheer magnitude of political issues that haunt such acquisitions, consider the challenge of integrating disparate, independently implemented systems from various vendors, with different code structures, nomenclatures, and technologies. How and where do you start? The challenge is daunting.

Applying the DIF Approach

I didn't plan to replace all the point-to-point integration code using the "big-bang" approach. Instead, my senior data architect, Jaime Gonzalez, and I established a conscious method that considered the following issues:

• Collapsing redundant processing
• Upgrading or replacing existing applications
• Addressing enterprise information needs.

Our intention was simple: Focus on the most pressing integration challenges in an effort to provide some immediate relief to IT and its user communities. The only concern was how to clearly identify, define, and rank process and data flows. Because this situation was a political minefield, we wanted to make sure that we were confident in our findings and our subsequent recommendation to the executive committee. The first model, shown in Figure 2, illustrates the overall steps we took to identify, define, quantify, and rank the project iteration opportunities. I have outlined the necessary steps as follows:

1. Antecedent documentation and known problems. Too often, consultants or IT personnel new to a project simply charge ahead - assuming nothing has been done or documented. Instead, the team must first fully investigate the existing documentation and interview IT subject matter experts (SMEs).

2. IT SME workshop. Users may see only one area of significant concern, but the real issues may be the result of other, less obvious components not visible to the user. So first conduct a joint application development (JAD) session with IT SMEs to get their perspective on what iteration opportunities exist. Invited participants nominate candidate processes and give brief explanations and tentative solutions. The SMEs should write each nomination on a flip chart and tape it to a wall for everyone to see. An example of a candidate nomination is:

• #1 Candidate
  Depreciation program and process is inefficient.
  
• Explanation
  The current process requires the rekeying of data between applications.
  
• Proposed Solution:
  Integrate G/L data for automatic feeding.
  

Also, you can enhance candidates with added information such as trouble calls or downtime trends.

3. Select candidate processes. Document and publish your candidate list. If the JAD session generated too many candidates, you can add a filter to reduce the number. For example: Can you complete a candidate within 90 days? This filter can quickly reduce the candidate list.

4. Get IT scores. Invite IT SMEs to complete the set of three surveys for each candidate in the list (see "The DIF Surveys"). The surveys identify the level of dysfunction (how ineffective and inefficient the process is), impact (how many other processes or applications the candidate affects) and feasibility (how likely are you to succeed in simplifying the process) of a particular candidate.

5. User workshop. After the IT SMEs have selected and scored their candidates, conduct a JAD session for your user community (UC) SMEs. The session begins with the candidates suggested by IT SMEs to which your users will contribute their own candidates. UC SMEs will complete DIF Surveys for IT as well as their own candidate nominations. You can then invite IT SMEs to score the UC recommendations.

6. DIF matrix. This matrix is the core of the model, which documents the survey results of candidate processes from each participant. Total the scores per participant and find the average based on the number of respondents. For more information on the matrix and scoring candidate processes, see the sidebars in the online version of this article.

7. Average DIF scores. When IT and UC SMEs have completed DIF surveys for all candidate processes, average the scores. Now each candidate process - whether introduced by IT or users - will have a single, overall DIF score. Potential project starting points will be those candidate processes that have the highest overall DIF score.

8. Select according to score. Rank candidate processes by their overall DIF score. However, the project leader can add a weight based on extenuating factors. For example, you can apply extra weight to all candidates that use available resources to complete. This feature gives project leaders some influence over the process.

9. Submit to management. In the final step, submit recommendations to executives based on quantifiable, statistically valid scoring. Decision makers now have a conscious approach for identifying a project starting point as well as laying out the subsequent iterations.

The DIF Surveys

Quantify the DIF perspective in a series of surveys referred to as DIF Score Sheets. Rate candidate process flows according to the following criteria:

Dysfunctional. How bad is the process in terms of effectiveness and efficiency? Important measurements include:

• To what degree does the time to complete the process exceed industry benchmarks?
• Is the net result of the process even used?
• How would you grade the quality (or reliability) of data in the end product?
• Are "redo"s necessary to create an acceptable end product?
• Do you have duplicate or redundant data in the end product?
• Does the end product need a significant amount of manual data entry?
• Is the data consistently available for the process?
• How much are user communities dependent on IT in order to access the end product?
• Do you have integrity problems in the stored data of the end product?

In a scale from zero to five, a zero would mean that a process is the least dysfunctional, whereas a five would mean that it is the most dysfunctional.

Impact. How many other processes and applications can you affect by making changes to the candidate process? Try to answer the following questions:

• If this process were completely redesigned, what percentage of the overall process would be simplified or reduced?
• How many applications or processes would you simplify or reduce by changing this process?
• How much end user pain (family time, quality-of-life complaints, nonmotivating work, and so forth) would you reduce?
• By how much would you reduce cost?
• What is the reduction in person-hours devoted to this process?

Feasibility. This criterion specifically attempts to quantify how likely you are to succeed in simplifying a particular process given the following:

• What is the probability of successfully carrying out a complete redesign?
• Do you believe that you have well-defined business rules to carry out the process?
• Does the necessary technology to accomplish a significant improvement exist?

A zero would mean that redesigning this particular process has the least feasibility, whereas a five would mean that it has the most feasibility.

You can add, change, or delete survey questions to suit your needs. However, it is important to retain the overall integrity of the survey process with regard to the goal: getting a DIF score.

The Next Step

As part of the DIF Matrix approach, we use the second model shown in Figure 3. The model establishes a formal procedure to complete the analysis of candidate processes. As illustrated, the model incorporates all traditional means to document and identify requirements such as use case, class diagrams, and interaction diagrams. The documentation lets you understand the process under consideration and determine how to best implement a solution.

The net of this model is one of three possible outcomes per candidate:

• Quick fix. Candidates you can quickly resolve.
• Redesign. Candidate processes that will require a full redesign.
• Gradual improvement. Much is still unknown about the process even after analysis. Consequently, you must approach the candidate process via a pilot project.

Take the High Road

Overall, the DIF approach effectively identifies, defines, and quantifies candidate opportunities and delivers clear criteria for executives to make decisions regarding a work schedule. This accomplishment is quite a coup because you often have numerous opportunities to consider in disparate environments with different agendas, business requirements, data sets, and technologies. The DIF Matrix lets your team stay above the fray.

Jaime Gonzalez, senior data architect and data modeler with The Focus Group Ltd., contributed to the content of this article and the development of the DIF Matrix approach.

Michael L. Gonzales, [mlg@starfocus.com], is the president of The Focus Group Ltd., a consulting firm specializing in data warehousing. He has written several books, speaks frequently at industry user conferences, and conducts data warehouse courses throughout North America.