Data in the Time of Cholera

How a 19th century physician's data warehouse helped prevent the spread of cholera

By Steven Johnston

Continued from Page 2

Data Purity

Another lesson to be learned is that the observational scientific method is much more important than the DBMS used to store your data. After all, most of the observational sciences got this far by using clay tablets and paper. In the observational sciences, data is treated with almost religious reverence. This stems from a strong master-apprentice tradition that is passed down through the generations from professors to their students. This tradition creates a solid ethical barrier against distorting data or drawing conclusions beyond what the observational data can truly support. The sanctity of the observational scientific method builds credibility, and this is a desirable quality for all data warehouses. There must be a strong element of trust in the data in a data warehouse for it to have business value.

Error Handling

The one thing I have not seen in data warehousing, which plays a key role in the other observational sciences, is the concept of observational error and error propagation. All scientific observations are subject to errors. Kepler was the first to recognize this problem in about 1600 when he was trying to work out a very accurate plot of the orbit of Mars. Kepler came up with the idea of "good enough for engineering purposes." This is a hard concept for people who deal with commercial applications. If you received a bank statement that said your account was $150,000 plus/minus $30,000, you would be very upset. However, if you were able to tell a CEO that John Doe will spend an additional $150,000 plus/minus $30,000 dollars with your company if you decrease delivery time by one week, you have delivered a good business value. Observational scientists have figured out ways to deal with observational errors by eliminating systematic recording errors, reducing random errors in the recording process, and increasing the signal-to-noise ratio of data in the data analysis step through various mathematical techniques.

Hire an Astronomer

Furthermore, it might be a good idea to add an astronomer, epidemiologist, or exploration geophysicist to your data warehousing team to bring in fresh analytical ideas. Many of the observational sciences have benefited from the cross fertilization of ideas.

Rate This Article Rating: 1 (best) 2 3 4 5 (worst) Comments: Optional e-mail address:

For example, the observational science of modern geology was invented by James Hutton in 1788 when he published his book, Theory of the Earth. Between 1788 and 1965, geologists mapped rock-formation outcroppings over a good portion of the earth and obtained a great deal of observational data from oil well bore holes. Analyzing this data allowed geologists to figure out what had happened over the past billion years of geological history.

However, by 1965 they were stuck. The geologists could not figure out what was creating mountains, earthquakes, or volcanoes. Then around 1965 some geophysicists began to analyze magnetic data collected by oceanographers who had dragged magnetometers behind their research vessels in the 1950s as they steamed across the oceans. This missing piece of observational data allowed the geologists and geophysicists to realize that mountains resulted when giant plates on the surface of the earth collided like a very bad car accident in slow motion. The resulting theory of plate tectonics also explained earthquakes and volcanoes and tied together all of the other puzzling observations that the geologists had accumulated over hundreds of years.

The greatest joy in being an observational scientist is discovering something that nobody ever knew before. It takes a lot of time and hard work to gather and process the data, and then there is a great deal of struggling in the data analysis step, but the rewards can be fantastic.

For further information on the remarkable work of Dr. John Snow, please explore the UCLA Department of Epidemiology Website created by Dr. Ralph R. Frerichs at www.ph.ucla.edu/epi/snow.html.

Steven Johnston (scj777@iols.net) is an IT architect at United Airlines.