Intelligent Enterprise

Cindi Howson and Mark Smith already weighed in with their impressions about MicroStrategy World and the impending release of MicroStrategy 9. Mark made a fairly complete overview of the product and proceedings and Cindi placed the offering in more or less competitive context, so I won't repeat either of those points of view (though I will take issue with Cindi on one thing – some of the other BI vendors may have been able to access multiple data sources in a single report, but not nearly as intelligently, efficiently or with more coherency than MicroStrategy version 9. These other approaches are ugly kludges in comparison).

I have a different perspective from Mark and Cindi, one that is borne of a history with MicroStrategy that goes back fifteen years. My review, with that perspective in mind, is this:

The data warehouse has been positioned as the sole source of analytical data in organizations, but that is changing. Rather than trying to remodel the data warehouse to accommodate fresher and more detailed operational data (near real-time activity in operational systems, process logs, etc.), these data sources will operate in parallel (or horizontally, whichever word you like) as complementary feeds to analytics. It takes too long and is too expensive to expand the data warehouse concept to do this.

BI tools like Microstrategy have to retool to be able to query multiple sources to satisfy a single query (they are doing that in the upcoming release 9, I believe). All of the other BI vendors will do the same.

In case you haven't heard it yet, here comes a new product category: Process Intelligence. But what does it mean? All of these terms overlap: Operational BI, Pervasive BI, Operational Intelligence, Process Intelligence, BAM, CEP (Complex Event Processing), Decision Management, Decision Services. Arguments over definitions tend to be vigorous for two reasons. First, the taxonomy of product classes tends to be pretty leaky and second, the stakes are so low.

The reason it is important to get some clarity on the definitions is that the wider BI industry (and I don't know what to call it) is driven by marketing, not by function or requirements. Software vendors invent things, acquire or get acquired by other vendors and give names to the combined capabilities they possess. Then it's packaged and sold to companies.

I don't hear this expression much anymore, but my wife used to use it all the time to describe the kind of chatter one can make about a subject, and seem knowledgeable, but possess only a very superficial grasp of it. In fact, she used to describe her entire academic career as having merely prepared her for cocktail conversation. I don't feel that way. I think my education, even three decades hence, was an excellent preparation for what followed, but then, I didn't study anthropology. LOL

I was on a panel at nGenera's (nee New Paradigm) Enterprise 2.0 get-together in Dallas last week. I missed the first day because I was speaking somewhere else and unfortunately missed listening to and meeting Ray Kurzweil, but the second day has some pretty good presentations by the nGenera staff, including Don Tapscott and my homey Nick Vitalari (it's amazing how you can live in a small town and only run into your neighbors at conferences).

This meeting is not to be confused with the much larger Enterprise 2.0 conference. The attendees are members of nGenera's network and gather a few times a year to listen to and present their progress on various research topics/projects at nGenera.

I got so many letters (isn't that a quaint way to say "email messages?") about the Hy Minsky posting that I wanted to pass along a fairly readable paper he wrote that spells out his Financial Instability Hypothesis.

Back to the topic. I listened carefully for the better part of two-and-a-half hours last week to Thomas Kurian, Sr. VP of Oracle, present the entire product set and positioning of Oracle's Fusion Middleware. He didn't crack any jokes and more or less stuck to the slide deck; nevertheless it was amazingly interesting. I liked almost everything I heard, especially the parts that were user-centric, such as unified metadata, common business semantics with shared logical models, single development environment, etc.

When it comes to analytics, Wall Street is clearly the leader. The best of the best head there after school to grab six-figure starting salaries. Some even see seven figures, based on their performance. They are the rara avises, the crème-de-la-crème, and whenever we speak about "Competing on Analytics," it goes without saying that Wall Street analytics represent the exemplar of what is possible for an analytic culture.

So why is Wall Street melting down?

When you sign up for a Webinar, or even just register to download a white paper, you can be sure that you will shortly get a follow-up phone call. The caller almost never has any inkling what you or their client does, so the questions are sometimes amusing, other times pretty dumb. I haven't gotten so old and cranky yet that this ruins my day, but I got a call last week that was notable.

After the brief introduction, the question was, "I want to ask some questions about your database."

"Excuse me," I said, "I'm an analyst."

"A what?"

There's been a lot of buzz lately about Google's MapReduce framework for speeding up the processing of large datasets. It makes you wonder, did Google just dream this up in last couple years while all of the database vendors were sleeping? Or, paraphrasing Isaac Newton, were they standing on the shoulders of giants?

The answer is, both.

Whenever the subject of business requirements for data warehousing and BI comes up, I try to bite my tongue because it's always at a time in the project when expectations are high and people are hopeful. I hate to rain on their parade, but this is one of those areas where best practices are often worst practices.

The idea that you can go "do" requirements gathering is canonical, but it's surprising and ironic how few practitioners actually believe in its value. This isn't a bias you want to expose to your clients, though. I find it particularly vexing that training sessions and conferences on data warehousing and BI usually have requirements gathering classes, taught by people who really ought to know better. I guess that's a subject for another day — why industry "experts" are content to disgorge training, for a fee, that they know is misleading, but is widely accepted.

Gary Cokins of SAS and James Taylor, my partner at Smart (enough) Systems, in an admirable attempt to disambiguate the terms Enterprise Decision Management (EDM) and Performance Management have, unfortunately, both gotten it wrong.

Gary claims that James "marginalizes Performance Management as being too narrow." Instead, he (Gary) suggests that "Performance Management and EDM are arguably very similar." James claims that EDM "goes one step further."

Although I don't remember when I first heard the term Services Oriented Architecture (SOA), I remember researching Web services around 2000. Back then, an architecture to handle Web services was unnamed, yet understood - at least to a degree. Now it has a name – SOA.

Back then, it seemed clear to me that Web services could provide more than just a way for Web-based applications to operate. With loosely coupled services communicating via standard protocol, while centralized directories allowed these services to describe their APIs, the sky seemed the limit. Reuse, long-chased but never achieved, seemed almost automatic. Platform independence, long-running transactions, and asynchronous processes — it would be like world peace.

The movie Ghostbusters is perhaps the single best training film for consultants I've come across. In simple words, they embody all the right stuff for a successful consultant which is, lets face it, a real craft, not just something to do between jobs. With motivational thoughts about teamwork, confidence, authenticity, client management and the projection of competence, these guys have it knocked. For instance:

Teamwork: There aren't many engagements where the success of the operation is dependent on just one player. It's important to organize for success and enhance everyone's contribution in a "whole is greater than the sum of the parts" mentality. That special esprit de corps that develops among small groups provides the energy to keep a difficult assignment on track. And, sometimes, teamwork requires splitting the team up.

• "We have the tools, we have the talent"
• "I love this plan! I'm excited to be a part of it!"
• "Let's split up, we can cause more damage that way"

Search isn't that great anymore. For one thing, it's become so commercial that it's really more like an ad search engine. SEO programs game the big ones to the point that you have to go to page 20 before you find something that isn't trying to sell what you're looking for. I want the Scotty Effect for myself (see my previous post). Why can't I ask a search engine questions and get sent to exactly the places with the answers, not 10,000 hits? Why can't the search engines help me assemble the information I need?

Tom Davenport suggests that the competitive playing field for businesses is analytics. I think we'd all be a lot better off if we could do some analytics for ourselves. What do you think? Here are some things I wonder about:

Do you remember the movie "Star Trek IV," when the crew needs to go back to the 20th century to find two hump back whales? When that movie was released, twenty-five years ago, we were already building pricing models with DSS software, we already had SAS to build models and do statistical work and we could write reports in FOCUS or any number of other tools. Compared to the things we can do today, this may seem primitive, but how different is it really?

Consider that the density of hard drives in the same period has increased five orders of magnitude, CPU speed even more so and the cost per unit of storage or MIP has fallen off the table. With that kind of improvement, a new BMW today would go from 0 to 60 mph in 0.00008 seconds, have a top speed of 15 million miles per hour and would burn gas at a rate of 2 million miles per gallon. Oh, and it would cost about 30 cents to buy. I haven't figured out the lease yet.

What's the Deal With Server Virtualization? Over the past few years, I've been suggesting that people stop managing from scarcity. What that means is that the cost of hardware has fallen so sharply that we should reevaluate our methodologies and designs that sacrifice function for resource efficiency. In data warehousing, we still create summarized versions of detailed data in order to avoid costly queries from eating up the big data warehouse, depriving people of the analytical content of the whole picture. In other cases, people are restricted to certain times of the day, or certain subsets of data or governors are placed on queries. With the cost of computing power dropping four or five orders of magnitude since data warehouses were invented, is this really necessary?

It's no mystery why diamonds are often referred to as "ice." Sure, they look like ice, all clear and faceted. But have you ever touched one and noticed it was cooler than you thought it would be? You probably thought it was just the power of suggestion, but it really is cooler. The reason is that a diamond's stiff crystalline structure actually shields the atoms from heat vibrations. So what does this have to do with information technology? A lot.

There has been a lot written about the suitability of relational databases in the ever-expanding Web world of text and pictures and video, even in Rajan Chandras' latest blog. Relational is given a lot of credit because of its staying power and incumbency, which is often confused with universal usefulness. But if you step back and think about it, there is nothing special about a relational database and in the world as we see it evolving, the physical structure, and even location of data, no longer matters. What made relational special was not the database, it was SQL itself.

Perhaps you noticed that I renamed this blog "Competing on Decisions" from "Addicted to BI." Now I'll admit that the latter had a certain anti-chic appeal with its allusion to substance abuse, but frankly, I'm recovering from my BI addiction, so it's time to move on. I've also come (slowly it seems) to the conclusion that informing people (BI) is part of an incomplete cycle. If we as a company make an investment in BI, it isn't the informing of people that matters, it's what happens next. The decisions.

Last year, James Taylor and I wrote a book, "Smart (Enough) Systems," to make the case for creating "decision services" in order to automate certain kinds of high-volume, low-latency decisions. Dubbed EDM for enterprise decision management, we laid out the sort of reference architecture for getting this done, which included predictive modeling, business rules engines, some form of either process automation, or at least a smooth handoff to operational systems and back end analytics to both evaluate the quality to the decisions and manage some form of adaptive control of the decision models (in other words, test new models and compare to the results of the existing ones).

It wasn't so long ago that if you were considering a data warehouse, your choices for a relational database platform were limited to Oracle, IBM, Teradata or Microsoft. In fact, I often wondered where all the choices went. Fifteen years ago, that list would have included lots of other choices. There were standalone database vendors like Sybase (their transactional database that is now Sybase ASE; Sybase IQ was not out in circulation yet), Informix and Red Brick. Most of the hardware vendors had their own offerings, too. We built some pretty good data warehouses for the times with Tandem, but there were also offerings from Digital, HP, Pyramid and probably a few others I've forgotten.

Database choices are now back, and then some.

There were lots of provocative questions and comments on my previous two posts ("Technology is Not the Driver of BI Adoption" and "BI and Technology: Part II"), so I thought I'd just batch all my responses together.

Thanks to all of you who responded to my last post with thoughtful comments. Rather than respond to each in a Lincoln-Douglas style debate, as Kurt Schlegel suggested, let me shift the discussion a little. Instead of arguing that technology alone can't move BI along, I'd rather explore the issue of what can.

To be effective, BI has to focus on simplicity of operation to achieve pervasiveness in the organization and beyond it. The model is the Consumer Web, which provides only the necessary presentation to perform the tasks at hand, and relies on open standards and loosely coupled services to perform the functions, which can be reconfigured dynamically. In the same way the users of the Consumer Web are willing to pay little or nothing directly (except for purchases), the cost of BI has to drop drastically from expensive, front-loaded perpetual licenses to pay-as-you-go on demand schemes.

I'm having some problems with a March 20, 2008 article titled "Gartner: Emerging Technologies Will Help Drive Mainstream BI Adoption." This has been the Holy Grail of BI vendors for over a decade — to increase the number of "seats" using their products, widely reported to be about 20 percent of an organization but clearly much less than that. What troubles me the most about this article, or rather, about Gartner's analysis, is the supposition that new technology is going to crack this old chestnut. It won't. There are only two pieces of enterprise analytical software (broadly speaking) that ever gained currency in organizations in the past two decades — Excel and Google. Wouldn't it be a good idea to understand why?

If you ever spend time as an administrator or even an editor on Wikipedia, you find that your initial enthusiasm for the concept wanes pretty quickly. I thought Wikipedia was a forum for interested people to present their knowledge in an open and influence-free environment, to be vetted by like-minded, optimistic people. As it turns out, it became a dumping ground for every crackpot, agenda, vendetta and misinformation-broker on the planet, which in turn, spurned a dizzying collection of Wikipedia policies and a subculture of enforcers of the policies. Obviously, there was a need to enforce these policies to eliminate all but the most carefully crafted articles, free of conflict-of-interest, lies, libel, etc., but the net result is that discussion of policy far exceeds discussion of substance today.

Harvard Business Review recently hosted a two-day conference in Miami called Think!Analytics (not to be confused with the firm with the same name sans the exclamation point) featuring Tom Davenport and Jeanne Harris, co-authors of the current best-seller, Competing on Analytics. Some of you may remember that I was pretty tough on Tom when the article of the same name came out in the Harvard Business Review in January, 2006, but we've since mended our fences; so much so that I went to Miami to join the festivities and Tom has agreed to keynote the Enterprise Decision Summit in October which James Taylor and I are co-chairing.

In "Measuring and Managing Performance in Organizations," Dorset House Publishing, 1996, Robert Austin made a very clear case that performance measurement often leads, paradoxically, to distortion and dysfunction instead of improvement. According to Austin — and I agree with him, having witnessed this phenomenon firsthand more than once — measuring an indicator of a performance (since we usually can’t indicate the actual performance itself), raises the risk of making things worse. How can that be?

Okay, I waited for the last shoe to drop. Now that IBM plans to gobble up Cognos, leaving only Microstrategy as an independent BI pure-play, and a much smaller one than Hyperion, Business Objects or Cognos, I'm ready to offer my opinion of the whole thing.

Yawn.

Who cares? All it means is that Business Intelligence software as we know it is a mature technology that finally got some attention from well-heeled giants like Oracle, SAP and IBM. Don't expect them to take these platforms and rocket them into the stratosphere. All they are after are the impressive customer lists and what we used to call in the commercial property and casualty business, gross line underwriting. One-stop-shop. Give me the whole deal.

Okay, so no one has used the term "MIS" (Management Information Systems) in years, but there was no way to work "IT" into the title with misinformation. I always used to laugh, by the way, when before the current CHIEF craze (CEO, CFO, CIO, CMO, CTO, etc.), the head of IT was often called the MIS Manager. The qualification for the job was, appropriately, being skilled in MIS management.

CIO Insight's October, 2007 report "How Valuable Is Business Intelligence to the Enterprise?" is another example of so-called research that makes no sense. The most curious aspect of this survey was that the respondents were all IT people. For my money, if you want to know how BI is doing, you should ask the people who use it (or don't use it).

Business Objects Labs is continually releasing new prototypes that can be downloaded and evaluated. You have to applaud this sort of approach, and some of these widgets are interesting and, as prototypes, indicate that there is a lot of creative thinking going on there. I think that Business Objects "gets it" sometimes, but then at other times, they have me scratching my head. They claim on the Labs Web site to deliver on, "the vision of 'ambient business intelligence (BI)' by allowing end users to access lightweight, secure, and personalized BI widgets at all times."

That isn't ambient BI, that's mobile BI. Ambient BI, according to my definition in January, 2006, should advise and drive businesses with embedded analytics, real-time decision tools and vastly improved capabilities for people and unattended processes in every corner of the organization, and beyond it. I suppose it's encouraging that Business Objects is investing in Web 2.0 capabilities, but their product offering as a whole feels a little like a strip mall — lots of things to offer, but no coherent thread running through it.

You can't swing a dead cat by the tail in this industry and not hit a story about exploding data volumes, service-oriented architecture (SOA), pervasive/operational BI and software-as-a-service (SaaS). Moore's Law is supposed to handle that first one, but can it really? And what about the others? Are we really ready for them?

Moore's Law, it's true, has driven the cost of computer hardware down, relatively speaking, but in one area, hot storage, there is a monster under the bed. While the density of disk drives has doubled every eighteen months or so, and the price per megabyte (really, of gigabyte now) has followed a similar pattern, there is a component of disk drives that is not electronic and doesn't follow the same trend. The actual data transfer rates have not been improving at a rate anywhere close to the increases in drive capacities. So we may have bigger, cheaper drives, but it doesn't mean that we can read or write more data any faster, or, at least not at the dizzying rate of increase of CPU's, memory and disk drive platters. That's the first problem.

Five years ago, I bought a covered wagon and put my family and whatever belongings we had into it and headed across the prairie. Well, figuratively speaking, anyway. I shut down my data warehousing/BI systems integration business and re-invented myself as a consultant to software companies. It wasn't really that bold of a move. I was tired of the constant travel and I had a fair amount of residual goodwill with the vendors so that I was able to pick up some work right away.

Along the way, I've learned a lot about how software companies work. And now, after five years, I'm something of a presumptive expert. From time to time, CEO's and founders and even venture capitalists ask me for my opinion about how to do this or that, and I always respond the same way: "I've never earned a dime running a software company, why would you ask me?" And it's true. I have the utmost respect for the entrepreneurs and managers who build something from nothing and have the attention span to attend to the details. I can't do that. I can help them in other ways, but the creation and nurturing is up to them. I'm not a company guy.

There is a lot of conflicting information about the term “Operational BI.” We need some research to sort out the jargon and propose a clear definition for the term (I’m willing). All of the following are being positioned as Operational BI (not a complete list):

• Data warehousing of operational data for reporting, with or without integration
• Replication of operational data for reporting
• Direct reporting from operational systems
• Federated reporting from operational systems
• Process Intelligence
• Inline integration
• Sensing applications
• Decision services
• Real-time BI

There are plenty of software moguls living in Santa Barbara, but after twenty years of living here, I haven't met very many. Perhaps it's because, for the first fifteen years, I wasn't here very much, or maybe it's just because I'm not a software mogul myself. So isn't it ironic that I had a very nice lunch a couple of days ago, in Santa Barbara, with one of the few software company CEO's from Santa Barbara I do know, John Patton, CEO of Sight Software. Ironic because he now lives in Hanover, NH.

I spent a couple days at the Data Governance Conference in San Francisco last week, not as a speaker, but strictly as a listener. It was nice of Davida Berger and Tony Shaw to invite me, and of course, it's always great to see friends and colleagues, but what I really wanted to get out of the three days was a deeper understanding of this concept of governance, what it means and how it works. I admit, I cringe at the word "governance," because it reeks of IT control and restrictions, something we've clearly had enough of for the past few decades. This, combined with the government's renewed interest in everything, seems like a perfect medium for the control freaks and compartmentalizers to push their agendas, creating more bureaucracy and less data democracy (though that is another term that makes me cringe, but let's save that for another time).

Some of you may have already seen a presentation floating around with a lot of WOW statistics about China and India and technology. If not, you can view it here: http://www.glumbert.com/media/shift

We're told that China has more honor students than the US has students, has the largest English-speaking population in the world, that Nintendo spends more money on basic research each year than the US spends on research into education and that a college freshman studying a technical topic will be learning things that are obsolete before he/she graduates.

I hope that the debate between Davenport and McAfee about Enterprise 2.0 was more enlightening than the excerpt I read, because I'm left with the impression that neither one of them gets it, and I'm pretty sure that isn't true. Web 2.0 is driving the way companies are doing business. For proof, look no further than the fact that VC money has virtually dried up for enterprise software over the past five years. The only true innovation going on now is at the edge of the Consumer Web.

I read with interest Tom Davenport's article, "Humans and Black Boxes" in the June, 2007 issue of BIReview. He raises the issue about whether humans are required in the analytics process anymore, given the offerings of vendors of unattended data mining tools. After all, with all of the hardware and bandwidth at our disposal, shouldn't systems be smart enough yet to swim around in the data and come up with predictive models that are more accurate than we mere humans can? Of course, Davenport doesn't believe that, and neither do I.

I don't typically watch C-Span, but I was watching last week because my wife was giving testimony before a Senate Committee. There were two panels, each with four panelists, and they included an Assistant Surgeon General, the head of the FDA Enforcement Unit, a senior person from the NIH who is in charge of the Women's Health Initiative, the largest (and most costly) clinical study of hormone replacement therapy ever, a Harvard professor of medicine and the head the Endocrine Society. How my wife managed to get to go last and get the last word is a mystery (though it's de rigueur around here), but that isn't what caught my attention. Here is what did.

These are extended responses to comments made in the original blog "Who Defines BI?"

Cliff Longman, CTO at Kalido, commented: "I think of BI as the car and data warehousing as the engine…Data warehouses should represent the historical view (and the "what if?" views as well if it is a business requirement) of data that a business relies on to judge its performance."

Cliff, we're pretty much in agreement. I think what we have is a problem of semantics (what a coincidence). We need to separate the data warehouse from data warehousing, which I think you did. The data warehouse is a repository of re-used information with historical context. Its use, going forward, will be diminished to some uncertain degree by advances in technology. It will not go away, at least not anytime soon. I have no quarrel with the data warehouse as a data source for reporting and analysis, but not as THE source.

I was more than a little surprised when I read the article "Think Critically When Applying Best Practices," by Bob Becker and Ralph Kimball. Unless I misread it, they have come around and defined BI as the total process, including data warehousing. This is something that the other prominent data warehousing guru's did a few years ago when, fearing they would miss the boat of the suddenly hot BI market, declared their IT-oriented data warehouse environment as BI. The fallacy in this is that the people who use BI were always conspicuously absent from the diagrams and descriptions of the data warehouse. Their architecture blueprints depicted "users" (and keep in mind that there are only two industries that call their customers users) as little stick figures crushed under the weight of their elegant, multi-colored architectures, or through demeaning models with names such as "Farmers."

Other analysts are preoccupied with the Oracle/Hyperion deal, but I'm not. There are so many important things to keep on top of that the financial doings of the industry are only secondary interests for me. After all, Hyperion itself is composed of three major companies – Arbor, Hyperion Software and Brio, some of which made major acquisitions of their own, including Brio's purchase of Scribe (SQR). This is just another step in an endless process. Some worry about consolidation of the market. I don't. There are dozens of companies waiting in the wings to become the next Hyperion.

Cindi Howson asked in a recent blog, "Given the perceived value of predictive analytics, why does it seem to have had such lack luster success to date? Like most things, I suspect the answer is part cultural and part technological."

I couldn't agree more. Predictive modeling isn't a crystal ball, and despite the efforts of Business Objects, Hyperion, Microstrategy and SAS to get predictive modeling into mainstream BI tools, there are a lot of other reasons for its lack of success. Knowledge Discovery is something that is best left to the experts, those with PhDs, years of experience or both.