Time for a Change

Traditional IT benchmarking does a poor job of addressing future needs

As the cost and importance of information technology grows in the organization, so does the need to justify new projects that add strategic or tactical value as well as expenditures on ongoing IT services. There are, of course, other traditional analytic tools such as ROI that corporations use for financial justifications, but benchmarking goes a long way toward giving senior management assurance and comfort with their IT investments. It is therefore important to view the benchmarking process as part of the IT service level.

Although it is easy to justify the “why” for benchmarking, it is much more difficult to decide “how” and “what” to benchmark. Even though benchmarking has existed as part of IT services for a while, it has not done a good job of developing a system that addresses the “hows” and “whats,” the reasons for which I’ll explain in this column.

The Traditional Approach

Traditionally, benchmarking IT infrastructure involves measuring the amount of raw inputs that go into delivering IT services. These raw inputs include:

•Human resources, which are typically evaluated in full-time equivalents (including such overhead costs as office space and so forth), and are somewhat broadly categorized, based on skills

•Hardware resources such as processors, memory, storage, and so forth, which have standard industry measures such as gigabytes and MIPs

•License fees for vendor software such as the DBMS and OS, which have a new measurement every few years, depending on the current trend. For example, the industry has gone from MIPs-based licensing in the old mainframe days, to usage-based licensing, and most recently, to seat management

•Contracts and consulting costs, which the HR category does not include.

As attention has moved from high-cost mainframe hardware into high-cost human resources, a shift in percentages has occurred among these inputs. Today, the emphasis is more on managing human resources (including productivity) than managing raw CPU power or disk capacity. It is also a good idea to know how much input resource you consume compared to your peers to produce the same unit of output.

But is that really all you want to know from benchmarking? The answer, of course, is “no,” for many reasons. When you benchmark your services in raw inputs, it does not reflect the value you add to those inputs. It is part of a value-add business in which the value is in the solutions you provide to business problems, using the resource inputs. Benchmarking using raw input measurements may tell you something about how you compare with your peers, but it does not help you much in further analysis or decision making. Consider how difficult it is to explain benchmark results to management in terms of your support costs per CPU MIPs and disk GBs.You can imagine their eyes glazing over.

Why do we benchmark this way, then? The answer is simple: Raw input is the most common denominator for peer comparisons. Therefore, many IT shops find this type of input the easiest to record and track in the benchmark database. As you move away from merely comparing raw input consumption, you approach the more difficult task of clarifying your true goals.

The Activity-Based Approach

What should be the correct measurement, then? As I mentioned before, you must measure in terms of activities in which you add the most value. In many activity areas within the organizations, interest has increased in activity-based costing (ABC) or activity-based management (ABM) in recent years. However, IT management has been slow in adopting the methodology for its own services.

Some examples of meaningful activities are ERP transactions in such heavily used modules as sales and distribution, email volume, and Web page hits. Comparisons with peers and other service providers in these activity terms make a lot more sense and are much easier to explain to management. Also, because these activities are business-driven, they are useful not just for comparison, but for projecting costs based on business volume.

You can’t attribute all activities directly to meaningful business propositions. In IT services, intermediary activities are worth measuring. This practice has become especially critical as IT services have evolved from the mainframe-based centralization to network computing-based recentralization of IT support services. Intermediary IT infrastructure services are now available and include systems monitoring, storage area networks, or enterprisewide network backup. These services are typically network-based client/ server architectures and support all business applications. It is critical to assess these intermediary IT infrastructure services, as the cost of setting up these services, as well as their operational maintenance costs, are substantial.

Two issues still remain unresolved. First, the point-in-time snapshot comparison with peers has value that is highly perishable in the fast-changing IT environment. Second, the benchmark at this level may tell you how efficient you are in your activities, but does not tell you how effective you are.

The solution to the first issue is doing continuous benchmarks, thus considering multiple points in time. This approach allows comparisons not only with peers, but also with your past performance. The problem, however, remains that this method is retroactive and does not tell you much about your ability to address future needs.

The second concern is how to determine whether your IT activities are producing the right results, ones that align with business objectives and strategies. In a fast-changing business environment, IT must always be prepared for changes. The key is to benchmark not only the absolute IT deliverables point-in-time, but also to look at your capacity for change. In mathematical terms, this strategy is like taking the first derivative of the point-in-time IT service deliverables — in other words, the rate of change in IT deliverables. Much research has been done in change management within organizations. IT departments are both the sources of change within organizations and are themselves affected by changes in other business environments.

Benchmarking your capacity for change will give you substantial insight into what you can deliver in the future. It is therefore a forward-looking, rather than a retrospective, approach. Furthermore, although your “capacity to change” can itself change over time, these changes happen at a much slower pace than the change in IT deliverables. You therefore do not need frequent benchmarking. Instead, you can focus more on the deeper issues that directly affect this capacity.

Benchmarking IT Capacity to Change

Your organization’s capacity to change relies on several factors:

IT staff skills. IT staff is now the most valuable asset in the IT organization beyond just physical IT infrastructure or applications investments. Unfortunately, current accounting practices do not provide corresponding treatment in human resource capital assets as they do for the physical assets. However, the main issue is not how many specialized staff you have in a particular technology. In fact, sometimes too many specialists in one particular technology are a hindrance to change, as the turf battles ensue. What you need is staffmembers who can learn new technologies and keep up with new developments. Of course, the organization must continue to invest in the staff’s skills and provide them with opportunities to learn. Investments in learning as well as the rate of qualification certificates or diplomas the staff has been receiving in recent years are good benchmark measures.

IT knowledge management. This area, even though it overlaps with IT skills, is worth mentioning separately. Some IT knowledge is tacit and you acquire it by doing, but another type is explicit and you can codify it. What are the organizational processes for IT knowledge management? How do you maintain the codified knowledge contents? What are the incentives for IT staff to share knowledge?

IT organization structure flexibility. IT organizational structures have evolved to keep up with the technology evolution. It has moved from hierarchical to network-based structures; self-managed teams are the norm. But these organizational structures do not just happen. They require a special culture and value system to form spontaneous cross-functional teams. A high level of trust among peers and self-discipline are two of the characteristics successful teams require. How many such teams have formed in recent years? Have they delivered results on time? What is the satisfaction level among members of the teams?

Project management. Change, of course, manifests mostly through completion of projects, whether they involve rolling out the new application or the next version of the desktop OS. These projects’ success rates therefore determine the capacity for change. Has IT been delivering projects on time? Have you delivered what you said you planned to deliver? How effective was your planning effort?

Financial management. As I discussed earlier in this column, you need to revamp benchmark matrices to be more activity-based. Ask yourself this question: Is the existing chargeback or other IT-funding mechanism tied to the business activity drivers? But beyond the activity-based costing, you also need to benchmark your ability to deliver new projects within budgets. What are the project costs’ control mechanism? Do they include all costs, or are some costs hidden?

Business process knowledge. IT has been one of the drivers of change in the business environment, but so are other factors like globalization and deregulation. As I mentioned before, IT is expected to be ready and flexible. An IT organization must understand the business strategy and organization’s strengths, weaknesses, opportunities, and threats. Is IT strategy aligned with the business strategy? Supply-chain management and customer relationship management make your organization’s perimeter walls less rigid and take IT beyond them. The new economy is presenting opportunities for all departments within the organizations: Finance, HR, marketing, R&D, and operations can form and reform different external network relationships. These network relationships allow the departments to provide functional support to the organization in a new way, but they heavily rely on the IT infrastructure.

IT operations management. Even though focus is on the capacity to change, you should not ignore the rigor of process disciplines in existing IT operations. Unless you have implemented disciplines in problem management, configuration-change management, or service-level management to create a stable operational environment, your chances of success in subsequent changes are substantially reduced. This reduction does not only take place because of technical difficulties in implementing further changes, but also because of a lack of credibility among an IT organization’s users and business management.

IT clients. We almost invariably focus internally on IT organizations in the benchmarking process. This scrutiny of course includes the client’s survey of IT organizations. The IT organization is, after all, a service provider. However, the truth is that the client needs to be able to absorb the changes caused by IT. Much like investments in IT staff skills, investments in client skills are worth monitoring. Also, the clients’ attitude and participation in IT projects is important. Do clients actively lead or participate in the IT project, or do they consider those projects something exogenous to their work?

Always Changing

This list is certainly not exhaustive for benchmarking the capacity to change. Benchmarking the capacity to change is not easily quantifiable and could lead to misinterpretation, if you don’t understand it well. But the returns are high, as you gain knowledge and understanding of the IT organization’s core. Without this knowledge, IT will continue to put band-aids on symptomatic issues, without ever tackling the root causes.