Finally, it’s important to develop tools and systems—ranging from business intelligence and analytics to customer surveys and focus groups—to capture the data that’s needed to measure success against a benchmark. Inaccurate or missing data can lead to wrong conclusions and poor decisions. In fact, it can derail an entire benchmarking initiative and weaken an enterprise. Moreover, as an environment changes, CIOs must be aware of how underlying numbers, ratios and percentages may change and lead to deceptive results. For instance, a metric such as “technology spending versus operating expenses” may spike in a down economy as revenues drop, but then appear low during a robust quarter.

Conventional IT metrics often center on server performance, applications, mips and other technical factors. But these measurements don’t necessarily tie into profitability or agility over the long term. Even executives who use more targeted IT-business metrics—such as “technology as a percentage of operating expenses” or “technology spending versus revenues”—risk missing the big picture. It may be useful to expand outward to general metrics such as Economic Value Added (EVA) or information productivity, both of which can help an enterprise understand IT investments in the overall fabric of the business.

Yet, even these criteria are likely to provide more of a snapshot than a complete picture. An IT department must understand where it fits in, says Forrester’s Cameron. Within some enterprises that operate in a regulatory environment or an industry that requires little more than system dependability and availability, a utility-based metrics model may fit the bill. At these firms, technology need only serve as an enabler.

At the other end of the spectrum, he notes, are organizations where IT must serve as a partner to spur technological innovation and help differentiate the business for customers and others. Not surprisingly, many IT organizations fall somewhere in the middle.

Complicating matters further is the fact that different IT systems and solutions may fit into any of these categories within a single organization. Problems typically develop when there’s a mismatch between what a department, business unit or entire organization expects and what metrics the IT department and the executive suite have in place.

For example, Harvard’s Kaplan relates the story of a British bank that nearly failed after it rolled out a Balanced Scorecard. Three years later, with the goal of building an IT platform that could be used anywhere in the world, the business was in shambles— despite exceeding all of its IT benchmarks. “There was a complete disconnect between the business needs and IT performance,” Kaplan says.