Research Results

By Terry Kirkpatrick

ROI 2002: How Do CIOs Figure ROI?




Most of the 404 top IT executives in our survey feel greater pressure than ever before to demonstrate a return on their IT investments. But there's no agreed-upon approach to calculating ROI. Worse, 70 percent believe their metrics don't fully capture the value of IT, and nearly half lack confidence in their ability to accurately calculate ROI. But IT executives who are more confident in their ROI methods also claim better returns on their IT investments, and they measure ROI more frequently and supplement commonplace ROI approaches with more sophisticated measures.

Six out of 10 IT executives say the pressure to calculate ROI is on the rise, while only 2 percent say it's decreasing.

There's no gold standard yet for calculating the value of IT. Seventy-seven percent of respondents cite cost reduction as one of their ROI methods; no other method was cited by more than 60% of respondents.

At 46%, nearly half of respondents are somewhat or not at all confident that the methods they're using are accurate.

For the IT investments we measured, 75% of respondents who do have confidence obtained the returns they expected or better. Only 56% of respondents with little or no confidence in their ROI methods made the same claim.

Those with confidence are also more likely to use more sophisticated ROI estimating techniques, and tend to find it easier to assess the ROI of various technology initiatives.



Oftentimes, the really important things are the hardest to measure.

"I spend more time on the intangibles than the tangibles," says Christopher Graham, vice president and CIO at the Church Mutual Insurance Co. in Merrill, Wis., of ROI measurement. "It's skewed about 60 percent intangible now and leaning more that way every day, because of the customer-service issues."

Like most insurance companies, Church Mutual is paper-heavy. Graham is determined to scan as much of this paper into electronic form as he can. The conversion will cost $1 million, but how will he know it's worth it? Some of the calculation will be easy. "What's it going to really save?" he asks. "Floor space? Paper? Paper storage costs? I can put numbers on these. But then you get into the intangible side—now I can have more than one person look at a document at the same time at any of our locations. This improves decision-making, communications and relationships with customers." Graham adds that he can take those same documents and attach them to the Web site so his customers can have some self-help options—they can print out a document and mail it in, for example. "But how do you put a dollar value on that?" he asks.

The CIOs who participated in our survey know this dilemma well. Seventy percent of the respondents, for instance, believe that monetary or business metrics don't fully capture the value of major IT projects. And 59 percent believe that non-IT executives don't place enough value on the intangible benefits of IT investments. All of this might not have mattered as much a few years ago, when budgets were less restrained, but 60 percent of our respondents say the pressure to prove ROI has increased in the past 12 months.

"There's been a real resurgence of interest in the whole value issue, of which ROI is a part, because people are being pressured to reduce their budgets and make sure they're working on things of highest value to the organization," says Mary Silva Doctor, principal consultant at Omega Point Consulting LLC, an IT management consulting firm based in Bensalem, Pa. "What we don't see, however, is just a focus on ROI—it's done in the context of contribution to strategic objectives. If a company has a well-articulated business strategy, you can usually distill that into a handful of drivers and ask, 'How does this project contribute?'" The survey bears this out: 51 percent of all companies and 62 percent of companies with more than 1,000 employees measure strategic value to determine the benefits of an IT investment.

Under the gun now, CIOs are questioning what they thought they knew about ROI. "I'm having a hard time defining what ROI is anymore," Graham says. "It used to be looking at a payback from a monetary standpoint. Today, other things have to be taken into consideration—will it be good for morale, increase the knowledge base of workers, facilitate better decision-making, improve communications, help with customer relations?"

Dr. Howard Rubin, executive vice president at Meta Group Inc. in Stamford, Conn., acknowledges the new complexity. "We're in a world where IT systems are now affecting things like relationships, and the results that show up are unstructured and intangible," he says. It's a cause-and-effect problem, Rubin adds: You know a new tool makes the sales force more effective, but showing its connection to higher sales and then to the bottom line is difficult.

It was thus no surprise to Rubin that nearly 46 percent of our respondents were either "not at all" or only "somewhat" confident in the accuracy of their ROI techniques. "Information systems today are more complicated, there are more intangibles, and it all flies in the face of computing ROI. That's the paradox of 2002: The pressure is on ROI just at the point where you can't define what it is. It's very hard to measure," he says.

All of the CIOs we spoke with were wrestling with how to link ROI to customer issues. Establishing that link is one of the challenges of CRM; 62 percent of our respondents consider ROI and other measurements of CRM initiatives to be difficult or somewhat difficult. The Sedona Group, a staffing firm based in Moline, Ill., has several sets of customers—the licensee staffing agencies across the country and the companies for whom they supply temporary workers. "When we try to do an ROI analysis on initiatives that would generate analyses or reports for these companies, some of it gets into gray areas," says CIO David Buzzell. "We can produce a new report, but what's the net gain? That's really hard to measure."

It's easier, he says, when it's a matter of customer retention, such as the time a firm using temporary workers supplied by Sedona wanted daily payroll figures by 10 a.m. "Basically, the client said 'If you can't do it, we're going to find a different vendor.' It's much easier to analyze the dollar value in this case."

In less obvious areas, though, it's also possible to take the measure of customer initiatives. "CRM is difficult for the same reason ERP was difficult: People have a belief that it's going to solve a variety of problems, and they're not necessarily focused on what the key questions are," says Sunil Subbakrishna, a vice president at Mercer Management Consulting Inc. "If your key issue is improving cross-sell rates, then you need to have a vision in mind for how you're going to do it and how CRM is going to help you. What items will cross-sell, and how are salespeople going to change their behavior?"

If quantitative measures aren't there, he says, companies can "at least try to describe in more detail what the qualitative results are going to be." Precise numbers are not as critical as taking into account such things as retraining your people, the capacity of employees to absorb a new process, the value of a project relative to other possible projects, and the risks of untried technologies, changes in the market and assumptions about how customers will react.

Then, Subbakrishna suggests, do scenarios—best outcome imaginable, expected outcome and the worst. "Evaluate those, and if you start falling toward the worst case you may want to stop the project."

At Church Mutual, Graham engages in a form of this, which he calls "outcome analysis planning." "If [a certain situation] happens, what are some possible outcomes?" he asks. His electronic file initiative will bring easily measurable process improvements. For example, the company can reduce claims adjusting expenses, which is significant: According to Graham, a quarter-point reduction saves the company $500,000. But the real value, he adds, will be in making better decisions.

"Some of the outcomes you can put a hard dollar amount on," he says, "and some you can't. So a lot of our decision-making deals with 'it's the right thing to do.' You get the consensus of the management team and you go for it."

Research Results

Research Results

The results are available in Adobe Acrobat PDF format. To download the free Adobe Acrobat Reader plug-in, click here.

  • RIO

Conclusion 01

Conclusion 01: Confidence

Confidence in their ROI metrics is strongly connected to whether executives regularly measure ROI. Without taking confidence into account, IT executives vary widely on what percentage of their budgets they evaluate using ROI, at which point during projects they estimate and measure ROI, and how long they are willing to wait for projects to provide a return.

Less confident IT executives calculate ROI for half or more of their budgets only 27% of the time, while more confident execs do so twice as often. Confident execs measure every IT investment 45% of the time versus 28% for less confident ones, and as part of strategic business initiatives 49% of the time versus 29%. These are strong signs that regular usage is linked to confidence.

Nine of 10 respondents estimate ROI when initially justifying an IT investment, but only 13% estimate ROI before a project is completed. That suggests that few IT execs readjust initial projections throughout projects.

Fifty-seven percent of it execs with more confidence in their ROI metrics measure the actual ROI of initiatives on an ongoing basis, as opposed to 32% of less confident respondents.

Conclusion 02

Conclusion 02: Methods

CIOs use a wide assortment of ROI methods, but measuring cost reduction is by far the most popular. Even so, there's no sign that any single method works well for all organizations and situations. However, IT execs who supplement traditional metrics with more sophisticated ones tend to have more confidence in their measures.

Cost reduction is the most commonly used ROI method, at 77%, followed by customer satisfaction, at 57%, and productivity improvement, at 56%. At 55%, IT's impact on profits and earnings is a more commonly used ROI metric than revenue gains, at 36%.

There are few differences between more and less confident execs in the use of traditional metrics such as customer satisfaction. However, execs with confidence are more likely to supplement traditional methods with such sophisticated measures as economic value added (43%), activity-based costing (44%) and internal rate of return (42%).

Just 59% of respondents believe they're using the right approach to assess IT's business value. But even the more confident execs have doubts: fully 54% of them feel their metrics aren't complete.

Seventy percent of more confident execs measure the benefits IT brings to the company's strategic goals, versus just 39% of less confident execs.

Conclusion 03

Conclusion 03: Involvement

How it's done, not who does it, is what counts when measuring ROI: There's no correlation between confidence in the numbers and who creates or evaluates them. Senior IT executives are the most frequently involved in measuring ROI, followed by senior business executives and the finance staff. The executives most CIOs report to—CEOs and CFOs—are the ones who most regularly review the numbers. Yet almost 60% of IT execs believe their non-IT colleagues do not appreciate the intangible benefits of IT.

Eighty-five percent of respondents report that senior IT executives are involved in measuring the business value of IT investments, with top business execs, at 59% and the finance group, at 49%. Organizations diverge most by size when involving mid-level IT managers: 51% of large organizations bring them in, but only 24% of smaller ones do.

Top executives are regularly involved in reviewing ROI figures before making IT investments. The CFO reviews IT investments before they are approved at 68% of companies, and the CEO reviews them, at 56%.

Fifty-one percent of respondents who have confidence in their ROI metrics believe non-IT executives place too little value on the intangible benefits of IT. That figure rises to 69% for res-pondents with little or no confidence.

Conclusion 04

Conclusion 04: Results

Intranets delivered the best ROI, while e-commerce and outsourcing disappointed most often. But across the board, IT executives with confidence in their ROI metrics reported less difficulty in determining ROI on different kinds of IT spending, even IT infrastructure, a notoriously tricky one.

Intranets met or exceeded the ROI expectations of 80% of respondents, followed by security, at 75%. The greatest disappointments were e-commerce, which fell below the ROI expectations of 33% of respondents, and outsourcing, at 32%. However, especially in technology investment arenas such as ERP, outsourcing and intranets, more confident execs more frequently reported favorable results than respondents with doubts about their ROI methods.

Confident execs regularly found it easier to assess the ROI of various initiatives than those that lack confidence. For example, 63% of more confident execs found infrastructure somewhat or very easy to calculate ROI for, versus 40% for those less confident. However, comparisons of other initiatives such as customer relationship management and security were less dramatic.

When asked to rate IT initiatives on the ease or difficulty of determining ROI and business value, the easiest were desktop systems (2.9 on a scale of 1 to 4), outsourcing (2.9) and e-commerce (2.7). The most difficult were knowledge management (2.0), CRM (2.2) and security (2.4).

Conclusion 05

Conclusion 05: Demand

While some IT executives look for extremely short time horizons for ROI on their projects, more than half will accept a maximum time horizon of two years or more. And expectations for the actual ROI delivered seem reasonable.

About a quarter of execs expect an ROI in a minimum of six months or less, though 23% don't use minimum periods to achieve ROI goals for IT investments. However, 62% will accept a maximum of two years or more before seeing ROI.

Respondents seem to have reasonable expectations for ROI. Only 25% of companies expect a 60% or higher return on their IT investments, and only 8% look for 100%.

More than three quarters of respondents either currently measure IT's contribution to strategic goals or plan to do so within the next year.



There's no question that IT execs are feeling greater pressure to prove ROI, and that many are attempting to do so. CIOs seeking to meet these demands should concentrate on developing the right ROI calculation methods. But which methods? Our recommendation: Take a page from those CIOs who feel most confident in their ROI calculations. As a rule, they subject a greater percentage of their projects to ROI analysis, and they measure ROI throughout the implementation of the project and on an ongoing basis after the project is completed. And they are much more likely to use such techniques as activity-based costing, economic value added, internal rate of return, and intellectual capital analysis. The result: They report significantly better returns on all of their IT investments.



How the survey was done: CIO Insight designed the ROI survey in partnership with Survey.com, a San Jose, Calif.-based supplier of custom research services. CIOs, chief technology officers, and vice presidents of information technology and services gathered from a number of sources, including third-party lists and other Ziff Davis Media publications, were invited to participate in the study by e-mail. The questions were posted on a password-protected Web site, and 404 qualified respondents replied from Jan. 8 to Jan. 14.

This article was originally published on 03-18-2002