Bumpy Ride

Data migration projects still plagued by problems

IE Index Dell Computer Inc. The company is still the market-share leader for Intel-based servers, says Gartner Dataquest, and is still steadily growing that share — by more than 3 percent year-over-year in Q4 alone. Enterprise Linux Companies Speaking of Dell, CIO Randy Mott pronounced Unix "dead" at LinuxWorld. The company has migrated 14 business-critical internal servers to Oracle9i Real Application Clusters on Red Hat Linux. Informatica Corp. Finally, signs of daylight: After a disappointing previous two years, the company turned a profit in Q4 2002. INDEX LEGEND HOT — On the upswing NOT — Possible trouble ahead STATUS QUO — No change

Companies spend millions of dollars and exhaust countless hours (above and beyond estimated time and costs) in their attempts to successfully complete data warehousing, CRM, BI, and other data-centric projects. According to The Data Warehousing Institute, data quality problems cost U.S. businesses more than $600 billion a year.

In an effort to improve data quality and to create the necessary information exchange resources required by today's challenging and dynamic business landscape, many companies have turned to extract, transform, load (ETL) and other related data quality tools. But how are these tools affecting project timelines and cost?

Information Architecture Team's (IAT) recent survey of approximately 300 developers, architects, and IT executives returned some startling results. IAT found that although most survey respondents have participated in several data migrations, they continue to be plagued by the same problems:

• More than 60 percent of respondents cited bad data and duplicate data as the primary reasons for data quality problems. Missing data accounted for nearly 19 percent of all data quality problems.
• More than 90 percent of respondents indicated that incomplete mapping and speed of data movement over network or load monitoring (or both) were the most critical problems resulting in project delays or failures.
• The median cost of a data migration project was 10.7 times the amount budgeted.
• Mapping the ETL project typically took nearly two months longer than planned; 27 percent of respondents say that mapping was finished within the scheduled time frame.
• Testing the ETL project typically took nearly 2.5 months longer than planned.
• Twenty-three percent exceeded their planned time for testing by more than 7 months.
• Nearly half of all respondents required additional hardware to exclusively support data migration activity at a median cost of more than $75,000.

While most survey participants were not unhappy with the ETL tools they used, some made comments that support the theory that ETL tools aren't the panacea that many think they are:

• "We need better impact analysis, automatic type changes propagation along mapping chains, and better and easier metadata to exploit."
• ETL tools "do nothing to improve the quality of [your] data."
• "ETL never addresses bad data and does nothing to improve the overall state of your data."