Oftentimes, the really important things are the hardest to measure.
"I spend more time on the intangibles than the tangibles," says Christopher Graham, vice president and CIO at the Church Mutual Insurance Co. in Merrill, Wis., of ROI measurement. "It's skewed about 60 percent intangible now and leaning more that way every day, because of the customer-service issues."
Like most insurance companies, Church Mutual is paper-heavy. Graham is determined to scan as much of this paper into electronic form as he can. The conversion will cost $1 million, but how will he know it's worth it? Some of the calculation will be easy. "What's it going to really save?" he asks. "Floor space? Paper? Paper storage costs? I can put numbers on these. But then you get into the intangible sidenow I can have more than one person look at a document at the same time at any of our locations. This improves decision-making, communications and relationships with customers." Graham adds that he can take those same documents and attach them to the Web site so his customers can have some self-help optionsthey can print out a document and mail it in, for example. "But how do you put a dollar value on that?" he asks.
The CIOs who participated in our survey know this dilemma well. Seventy percent of the respondents, for instance, believe that monetary or business metrics don't fully capture the value of major IT projects. And 59 percent believe that non-IT executives don't place enough value on the intangible benefits of IT investments. All of this might not have mattered as much a few years ago, when budgets were less restrained, but 60 percent of our respondents say the pressure to prove ROI has increased in the past 12 months.
"There's been a real resurgence of interest in the whole value issue, of which ROI is a part, because people are being pressured to reduce their budgets and make sure they're working on things of highest value to the organization," says Mary Silva Doctor, principal consultant at Omega Point Consulting LLC, an IT management consulting firm based in Bensalem, Pa. "What we don't see, however, is just a focus on ROIit's done in the context of contribution to strategic objectives. If a company has a well-articulated business strategy, you can usually distill that into a handful of drivers and ask, 'How does this project contribute?'" The survey bears this out: 51 percent of all companies and 62 percent of companies with more than 1,000 employees measure strategic value to determine the benefits of an IT investment.
Under the gun now, CIOs are questioning what they thought they knew about ROI. "I'm having a hard time defining what ROI is anymore," Graham says. "It used to be looking at a payback from a monetary standpoint. Today, other things have to be taken into considerationwill it be good for morale, increase the knowledge base of workers, facilitate better decision-making, improve communications, help with customer relations?"
Dr. Howard Rubin, executive vice president at Meta Group Inc. in Stamford, Conn., acknowledges the new complexity. "We're in a world where IT systems are now affecting things like relationships, and the results that show up are unstructured and intangible," he says. It's a cause-and-effect problem, Rubin adds: You know a new tool makes the sales force more effective, but showing its connection to higher sales and then to the bottom line is difficult.
It was thus no surprise to Rubin that nearly 46 percent of our respondents were either "not at all" or only "somewhat" confident in the accuracy of their ROI techniques. "Information systems today are more complicated, there are more intangibles, and it all flies in the face of computing ROI. That's the paradox of 2002: The pressure is on ROI just at the point where you can't define what it is. It's very hard to measure," he says.
All of the CIOs we spoke with were wrestling with how to link ROI to customer issues. Establishing that link is one of the challenges of CRM; 62 percent of our respondents consider ROI and other measurements of CRM initiatives to be difficult or somewhat difficult. The Sedona Group, a staffing firm based in Moline, Ill., has several sets of customersthe licensee staffing agencies across the country and the companies for whom they supply temporary workers. "When we try to do an ROI analysis on initiatives that would generate analyses or reports for these companies, some of it gets into gray areas," says CIO David Buzzell. "We can produce a new report, but what's the net gain? That's really hard to measure."
It's easier, he says, when it's a matter of customer retention, such as the time a firm using temporary workers supplied by Sedona wanted daily payroll figures by 10 a.m. "Basically, the client said 'If you can't do it, we're going to find a different vendor.' It's much easier to analyze the dollar value in this case."
In less obvious areas, though, it's also possible to take the measure of customer initiatives. "CRM is difficult for the same reason ERP was difficult: People have a belief that it's going to solve a variety of problems, and they're not necessarily focused on what the key questions are," says Sunil Subbakrishna, a vice president at Mercer Management Consulting Inc. "If your key issue is improving cross-sell rates, then you need to have a vision in mind for how you're going to do it and how CRM is going to help you. What items will cross-sell, and how are salespeople going to change their behavior?"
If quantitative measures aren't there, he says, companies can "at least try to describe in more detail what the qualitative results are going to be." Precise numbers are not as critical as taking into account such things as retraining your people, the capacity of employees to absorb a new process, the value of a project relative to other possible projects, and the risks of untried technologies, changes in the market and assumptions about how customers will react.
Then, Subbakrishna suggests, do scenariosbest outcome imaginable, expected outcome and the worst. "Evaluate those, and if you start falling toward the worst case you may want to stop the project."
At Church Mutual, Graham engages in a form of this, which he calls "outcome analysis planning." "If [a certain situation] happens, what are some possible outcomes?" he asks. His electronic file initiative will bring easily measurable process improvements. For example, the company can reduce claims adjusting expenses, which is significant: According to Graham, a quarter-point reduction saves the company $500,000. But the real value, he adds, will be in making better decisions.
"Some of the outcomes you can put a hard dollar amount on," he says, "and some you can't. So a lot of our decision-making deals with 'it's the right thing to do.' You get the consensus of the management team and you go for it."