Utter the word “metrics” to any cio and you’re likely to get an earful about opportunity and risk. It’s a fairly simple concept: Measure the right things, and the business will likely thrive. Monitor the wrong factors, and the same enterprise will veer off course and perhaps even crash.

While no sane executive would ever dispute the validity of metrics and benchmarking, figuring out how to apply them can prove daunting. And nowhere is this more obvious than at the chaotic intersection of business and IT.

“It’s very difficult to identify a benchmark that is actually relevant,” says Bobby Cameron, Forrester Research’s principal analyst for CIO issues. “So much of IT’s performance and budget is defined by the business and technical context in which spending and investments occur.” Too many organizations, he says, get lost in a tangle of measures and numbers that may or may not prove meaningful.

“The key is to measure IT’s value to the business,” Cameron suggests.

It’s an argument that’s tough to dispute, particularly in today’s brutal economy. However, the path to success can prove remarkably bumpy. For one thing, no single set of benchmarks works for every organization—or even for the same organization over time. Attempting to apply a set of industry best practices or use a preset Balanced Scorecard or Six Sigma template can result in abysmal failure. “More than a few companies have gotten lost by chasing other companies’ metrics,” says former CIO Dan Gingras, now a partner at consulting firm Tatum. “Benchmarking must be internally rather than externally focused.”

For another, “It’s easy to fall into the pathology of choosing metrics that indicate success while an organization is actually struggling,” says Harvard Business School Professor Robert S. Kaplan, co-creator of the Balanced Scorecard. “If you’re too inwardly focused and you think mostly about IT as its own function, you can easily wind up measuring the wrong things.”

Consider: A bulletproof e-mail system and five-nines availability may allow the IT department to toast its performance, but if customer satisfaction is lagging or revenues are sagging, the metric is useless.

See also:

Ruby Tuesday CIO: Feasting on Metrics

2nd Wind CIO: Exercising Measurement Options

Harvard Business School's Robert Kaplan: Culture Code