ROI 2003: Do You Have Any Faith In Your ROI Numbers?

By Allan Alter  |  Posted 03-17-2003

ROI 2003: Do You Have Any Faith In Your ROI Numbers?

68% of CIOs measure the benefits technology brings to strategic goals

76% of IT execs believe the pressure to measure intangible benefits has increased

73% of CIOs don't calculate ROI on projects after they're completed

70% of companies find it difficult to calculate ROI

44% of the results of ROI calculations are subjective

Given the state of the economy and the penny-pinching mood of many corporations large and small, it's no surprise that the pressure to measure and improve ROI continues to increase. Judging by this month's research study, however, the techniques most CIOs use aren't up to the task. According to the more than 375 CIOs who responded at the end of last year to our second annual ROI survey, more projects are being assessed with various ROI metrics, more business executives are involved in evaluating the potential return on IT projects, and more companies are trying to tie the benefits of IT more tightly to their strategic business goals. The downside: despite their good intentions, fully 70 percent of IT executives concede that their metrics don't adequately capture the business value of their projects; indeed, half aren't sure their companies use the right approach. That would explain why less than half of respondents bother to calculate returns on such critical systems as ERP, corporate portals and business intelligence. All in all, the state of the art of ROI practices is dismal.

Research Findings

Research Findings

The results are available in Adobe Acrobat PDF format. To download the free Adobe Acrobat Reader plug-in, click here.

  • ROI


: How do executives match metrics to their companies?">

How do executives match metrics to their companies?

The most common technique companies use to measure the value of IT investments isn't a standard accounting measure such as net present value or a trendy metric such as balanced scorecards. It turns out that "company-specific measure"—any internally devised measure—handily beat these kinds of measures, and even edged out the standard ROI formula—(savings + additional revenues)–cost—according to 56 percent of respondents to this month's survey on ROI techniques. Company- specific measures also scored high for accuracy. What makes them so popular?

Typically, company-specific metrics score points with simplicity. "Complex measures are difficult to explain or defend," says Rajiv Banker, chairman of the accounting and information management department at the University of Texas at Dallas. Even in the financial services industry, few executives "are thinking fancy around these accounting schemes," says Christopher Dallas-Feeney, a New York-based vice president with Booz Allen Hamilton.

Rather than relying on one metric, says Banker, most companies create a portfolio of measures. The respondents we spoke to bear this out. While some judge new systems by a single, simplistic yardstick or take an ad hoc approach, about half of the nine IT executives we spoke to supplement simple cost/benefit formulas with other measures, or use several measures in tandem, often tweaking them to account for the cost of capital.

In 1998, Chase-Pitkin Home & Garden, the Rochester, N. Y.-based home improvement division of Wegmans Food Markets, Inc., quit trying to quantify intangible benefits and turned instead to measuring hard savings. That turned out to be inadequate, says CIO and corporate controller T. Christopher Dorsey, because it didn't take the cost of capital into consideration. So today, Dorsey has added calculations of economic value-added. "It helps the IT group understand that for every resource we take for a project, there's a cost," says Dorsey. The company also sets payback periods: longer for spending on maintenance and upgrades, shorter for systems designed to improve productivity or customer service.

Like Chase-Pitkin, the Dow Chemical Co. combines several standard value measures, but the Midland, Mich.-based company's approach is more rigorous. According to Paul Janicki, Dow's finance director for information systems and e-business, the sponsors of a new system must calculate benefits using net present value, determining benefits above cost on a year-by-year basis taking inflation and interest rates into account. They must also plot out how much cash their projects will need yearly, assess the risk of the project and consider ways to mitigate risks. Finally, says Janicki, executives compare the project to others under review in order to determine which has the most potential value and fit with the company's business and IT strategy. Only then does a project get a green light. After implementation, the system's performance and cost is benchmarked against similar systems installed by other companies.

IT executives also devise their own ways to demonstrate intangible benefits. Financial managers at Navigant International Inc., a firm that helps companies manage travel, were unconvinced by claims that giving customers a digital dashboard to track key travel data would bring bottom-line benefits, according to CIO Neville Teagarden. So he and other managers formed an advisory board of 10 customers and discussed the digital dashboard idea with them. They did the same thing with a dozen potential new customers. The feedback was uniformly positive, and the project got the okay. After reactions to a prototype confirmed this early feedback, the system, called AlertFLYR, went live at the end of December.

Given how broadly IT executives define "company-specific measure," it makes sense that it should rank high in our survey. But consultants, particularly those who provide ROI services, note many potential dangers. If developed in isolation by the IT organization, these metrics risk being tied to internal IT performance rather than business value, or being perceived by outside executives as self-serving. There's also the danger that such a metric is incomplete, or used inconsistently by different parts of the business. CIOs may trust the measures they devise, but "if you ask company management if they trust these measures, you'd get different answers," says Christopher Gardner, partner and cofounder of iValue, an IT strategy firm based in New York.

To avoid such problems, CIOs should collaborate with non-IT executives in the creation and use of these metrics, and follow up initial ROI predictions by measuring after the project is complete, according to Howard Rubin, a Meta Group vice president. Ultimately, making sure any metric is accurate requires a relentlessly skeptical attitude. "Is this measure really getting at the value? Am I getting all the inputs right? Am I assessing risks properly? Asking questions like this is the key," says iValue's Gardner.