Roundtable: The Trouble With ROI

By CIOinsight  |  Posted 11-15-2003

Roundtable: The Trouble With ROI

Something is wrong with this picture. In December 2002 CIO Insight surveyed 378 IT executives on their attitudes and practices regarding return on investment. The headline of the results we published in March 2003 sum up our findings: "Do You Have Any Faith In The Numbers?" Even though 60 percent of the respondents said the pressure to demonstrate ROI had increased in the past 12 months, and 76 percent said the pressure to place a dollar value on intangible benefits had increased, many IT executives found ROI to be a frustrating and difficult exercise. Fully 70 percent conceded their metrics don't adequately capture the business value of their projects; indeed, half weren't sure their companies used the right approach. Many simply distrusted their own numbers: 46 percent did not believe their methods were accurate, and 44 percent believed the results of their ROI calculations were subjective.

We were intrigued to find CIOs so dissatisfied with a metric that they are under increasing pressure to use. It led us to wonder just what's causing so much discontent—indeed, whether the concept of ROI needs to reconsidered, or even abandoned in favor of a different way of looking at the value IT brings to companies. And so CIO Insight Executive Editor Allan Alter, on August 11, convened a discussion at the magazine's editorial offices to discuss the problems and limitations of ROI. Participants included CIOs, financial executives and business management experts, including an IT executive who refuses to use ROI at all. Are there viable alternatives?

The following is an edited transcript; For the full transcript please click here.

CIO Insight: Is the often elusive quest for ROI a symptom of a bigger problem, like a lack of trust between the business side and the IT side?

Barkley: One of the former presidents of our company was always fearful of "the system that ate Chicago." So yes, you're right. There's a fair amount of mistrust of the financial analysis capabilities of information technology people. And there's a fair mistrust of the technical competence of the financial people.

Murphy: The trust issue goes beyond IT and finance. Our finance group requires ROIs for any capital expenditure beyond $150,000, and that includes building a new ship. Once I, as CIO, have established that I have a methodology and my team has process and discipline for doing these things effectively, however, that changes the conversation. That's where the trust comes. But that's no different from any other business unit.

Hirji: I think IT professionals probably brought this upon themselves in some way. When I was a consultant, it was not uncommon to have projects between $200 million to $500 million in size, most of which failed. They got three-quarters of the way through implementation, but the systems didn't get turned on. So it was easy to calculate ROI on those—zero.

What's the problem here? Poor leadership?

Riazi: They say the life of a CIO is three years, because the first year is infrastructure, the second year is begging for money to do ERP and the third year is when you actually do it. And by the third year, you have the whole organization ready to shoot you. You learn as a CIO how to separate those who hate you from those who are undecided. How do you get ROI? I think we start looking at our failures, because our success has been abysmal. Our credibility is really shot. And that's why I refuse to go to see my CFO with an ROI case; he's going to laugh at me and kick me out of his office. [Laughter]

So how does any CIO survive?

Riazi: I don't use ROI. I don't think CIOs should use ROI because the "R" isn't in my department; the "I" is in my department. The R is in the business, over which I have absolutely no control. So why should I use it?

Cameron: I disagree. The bottom line is not that ROI fails or that the projects fail or that maybe there's a bad odor of IT history. It's that we in IT have tried to do these things alone. The cultural change is for us to become part of the business so that we relate to the rest of the firm like a manufacturing plant general manager does.

Fletcher: You need to get over that sort of silo thing. The R and the I all end up in shareholder value.

Gardner: I don't think that we can escape this question of how we measure the value of IT, because management will always have a problem with IT as long as you can't measure its value.

Cameron: Is there a value for IT? I would argue there isn't. There's a value for business execution or delivery or value delivery with IT as a component.

Gardner: Which is fine. You've got to look at what IT is doing to contribute to the overall value of the company. I think there's also an issue around precision. I remember seeing a vendor ROI analysis that came up with a number like 3,287 percent for some system. That's just not the way to think about these things. And you need to look at risk and return together, and risk is completely left out of the equation. For those things that you can't quantify and that are very uncertain and ambiguous, one can at least make an estimate of the risk. Is it something that could affect the entire success of the project, or will it have a negligible effect? You can at least do that kind of thing.

Kalafut: We know there is no magical power in the technology itself. Erik Brynjolfsson and his colleagues at MIT have studied companies' investments in IT and found that they're almost invariably accompanied by much larger investments in intangibles such as training and by the empowerment of employees as well. All of it actually comes down to how you empower people to use those technologies rather than the technologies themselves.

Atti, are vendors part of the problem?

Riazi: Yes. They come in, they sell you a concept that just looks terrific and wonderful, and they promise the ROI without sharing the issues related to change in the organization with the CFOs and the CEOs. We are discussing an illogical issue in a very logical way. We are talking about people, and when you implement technology, you change the way they work. ROI fails because we forget these are people who are going to worry about their jobs and their training and experience, and wonder what's going to happen to me. Hundreds of people have lost jobs, but that's considered irrelevant. Between the vendors and us CIOs, there is a responsibility to look at what happens when technology changes the way an organization works. But the vendors have underestimated the importance of that. They say you put this ERP in and it's going to reduce head count and make you more efficient and better. You put CRM in and you're going to automatically increase customers. It's not real.

Schwartz: There are really two aspects to how vendors use ROI. One, they're trying to make a compelling value proposition to get the CIO's attention. Two, they're trying to help their clients and customers create an internal justification to build a business case. Some vendors are going to do this very well, and others are not going to do this very well at all. They're learning as well.

Fletcher: It seems to me that it's very difficult to expect the vendor to have an understanding of what is going to drive the return on investment. For one thing, the discount rate that is used on the return on investment must reflect the capital costs of the company, not of the vendor. Second, I don't know how a vendor could possibly understand all of those soft, cultural costs that Atti was talking about. It's hard enough for internal corporate management to assess those costs.

Schwartz: Vendors will never understand their customers' business the way their customers do. You know your businesses inside and out. But what vendors can do is bring a very valuable outside perspective and perhaps some other experience, some benchmarks, some research on what other companies have achieved.

Riazi: I want a vendor that understands my business and understands my problem. It's their job to understand it if they're going to sell to me. I rarely get a vendor in my office that says, "I understand you're facing this problem in the advertising business, and I believe this could address this part of the problem. It's not going to be easy to put it in. I can promise you it will be hard. And the users may not want it because it's going to change the way they work. But I think it's the right solution." That is so much more genuine than a person who just comes to me and says this is going to give me a competitive edge. Partnership is crucial to the success of ROI.

Murphy: I cannot imagine such a scenario where we would have a project go forward at Royal Caribbean where I was developing the ROI. It isn't done. The business unit has to be accountable for the return on investment, except for a few real core infrastructure-types of initiatives, and even then I can show the value to the business.

Barkley: We require that the business case on every project over $1 million comes to my desk. Before it gets to my desk, it's signed off by the business CFO. Even after he signs it, I'll call him up and ask, did you understand what you signed? Do you understand how you're going to find a savings? Because I'm not the guy who's going to look for them, you are. We require that.

Okay, so despite problems around the concept of ROI, it seems like ROI is still something people need to measure. Is that right?

Barkley: Maybe we're overly focused on ROI. Here's why. At our company we suddenly realized, in looking at our budget process over the past two years, that we spend 85 percent of our time on 30 percent of the budget. Projects are important, capital investment is important, but 68 percent of our money is to keep the lights on and the business running. We hadn't done a good job at budgeting and analyzing that. We had to figure out a way to do a better job of budgeting "keeping the lights on" spending, and aligning those systems with the strategic intentions of the company. So we've actually changed our budget process where we do the lights on budget first. Then late in the budget process, we get the project budget from the businesses and go through a strategic alignment value analysis where we measure, among other things, internal rate of return, business value risk and dependency.

Fletcher: A couple of thoughts. ROI is a good tool, I think, as long as it is used consistently throughout the company for all kinds of different decisions, whether it's marketing, HR or anything else. You can't just say IT goes the ROI way but marketing does its own spiel. That doesn't work. Second, I think ROI is a very useful tool for allocating funds if you have a lot of demand for those funds, and you have to make a judgment call between one versus another. If you don't have that kind of situation and you have all the funds that you need in the world, I think free cash flow is a much better tool. It too, though, is subject to lack of specificity or subjectivity because, let's face it, we're all human and we all make projections from right now. If you ask people to make a projection now about revenue growth, their projection would be very different from anybody's projections three years ago. I think the most important thing is always the strategic driver. Why are we making this investment? If I have only a $100 million and I have to decide whether I'm going to go with Atti's project or I'm going to go with a marketing guy's project, I've got to make some kind of an allocation decision. But if I don't have that challenge, as long as it's free cash flow positive, let's go do them both.

Bobby, is looking at IT investments the way investors look at portfolios a helpful way to look at and think about ROI?

Cameron: Let me make a couple of comments before I answer that. One is that it really is about the process. If the process doesn't involve business and IT equally, nothing works. Second, we've focused on ROI as if it was the be all and end all, but not everything can be justified. Neither mandatory nor biggest-customer-tells-me-I've-got-to investments are justified based on ROI. Return on investment is really calculated for a portion of the investment stream that is justified based on changes to operations. There are firms committed to ROI that also justify projects based on softer metrics. EMC Corp., for instance, uses soft metrics for things that might improve customer satisfaction, but would be very difficult to prove an ROI.

That lays the ground for what I'm about to say, which is that most firms want to invest to drive competitive advantage and improve stakeholder value. Frequently, firms do their darndest to understand what the strategies are, and what they need in order to accomplish these goals. The problem is there's usually no linkage between that planning and investments in the prioritization process. What I've seen emerging is, in fact, a portfolio mechanism, meaning we look at everything as a group of investments and we compare them to each other, and that the process attempts to adjust priorities to strategic needs. But there is no correlation in the data we've picked up between having a portfolio and success at prioritization.

So what are the two or three things that companies need to do first and foremost?

Cameron: The best firms start by committing capital investment dollars in specific buckets. For instance, if I have $100 million to invest in capital projects, I say 60 percent is going to customer value, 30 to value creation (manufacturing or back-office systems) and 10 to enabling. By breaking them into buckets, you start to clean up the conversation. Then, those buckets have specific business goals they're trying to achieve. Improving employee productivity might be a good one in the value creation. Then the third step is to prioritize projects based on their impact on those objectives, some of which are ROI and some of which are soft.

Denise, you were a CFO. Would this approach pass muster on the business side?

Fletcher: Well, I start off with shareholder value. That is the driver, nothing else is. Then the next step is, what drives shareholder value? The only thing that studies have shown that actually, truly correlates to shareholder value is free cash flow.

In my view, the key is not bucketing or anything like that. The key is to start your process with a strategy. Where are you taking the company? Once you've identified where you're taking the company, the next step is to identify the strategic drivers that are actually going to shift the company to where you want it to go. Then you look at the strategic drivers and you ask, what IT, what infrastructure do I need to make the strategic driver happy? And if I can't swing that, I obviously have to get rid of some of the things that I've been doing until now. As I see it, the benefit of going through a process like this, where you identify the strategy first, is that IT flows in as an enabler of the strategy, not as somebody trying to implement a specific thing.

Hirji: I think the cardinal rule has to be to keep it simple. Anyone can take something and make it overly complex. It looks very precise, and tons of man-hours get chewed up doing it. Everyone feels better, but the reality is you're no further ahead. So the real genius is making decision-making simple.

Let me walk you through how we do it. We have specifically identified what our strategy imperatives are. We have very clear goals and we can align all the projects against them. If it doesn't meet one of the goals, it's off the table. Next, every week the business and technology folks get together and run through the prioritization, and they list it one through ten in terms of what they want. We do that weekly because our projects are small and the marketplace changes; what was important last week may not be important this week. Before any project gets on that list, it has had a business case developed, which has a number of things in it that say this is why it's important. Sometimes it's ROI; sometimes it's a bunch of things. Whether it's ROI or not is irrelevant. The important point is that that process gets both the IT folks and the business folks to say we jointly believe this should be on the list, we think it should be number X out of 50 on the list. It's irrelevant what the number is. What's more relevant is getting the alignment.

Fletcher: The thing about ROI is this: The input is critical to the output. Let me explain. I was CFO of MasterCard until a little while ago. One of the major investments that we decided to make was in brand. Investing further in brand meant that we were going to get more response from customers, and to deal with that we needed to add storage capacity. If I look at my investment in brand and include in that my storage capacity, it will give a very positive return. If I look at storage separately, it is not likely to give me a very high return. So I think the most critical thing is investment with the overall strategy of the company. I think ROI is only a tool, and the tool is only as good as the input that goes into it and as good as the boundaries that are defined around it.

Atti, does this begin to address your concerns about bringing back the human element when looking at value?

Riazi: I'm not certain. I've heard a lot of great ideas here. But we always fall back on our comfort zone, and I think we are falling back to a logical approach to a problem which I find to be emotional. The best way to make ROI work is to look at an organization organically. You have to look at it strategically, look at it organically, and acknowledge IT will help, but ROI shouldn't drive it. The reason ROI has failed miserably is because it's been sold as a driver, and that's when it becomes very difficult for it to deliver. I think we have sold technology to the business in isolation. Here it is, you put it in and it's going to be wonderful. I call it the diet pill: You take it, you lose 50 pounds and you don't have to worry about anything else.

Let me share one story about my 8-year-old nephew. I went to buy him a Halloween costume; he wanted to be Superman, so I bought him a Superman costume. I took the costume home and I looked in the back and it said: "This does not enable you to fly." [Laughter] That's an important lesson for us. The message out there is all this wonderful technology stuff is going to make you fly. It doesn't, and it won't. Somehow we need to come to terms with that.


's Wrong With ROI?">

What's Wrong With ROI?

  • It's hard to separate the ROI of IT from the ROI of a business process or project.

  • ROI gives the illusion of precision and objectivity.

  • There's no universal definition of ROI.

  • ROI formulas don't account for company cost structures, economics or risk.

  • Few executives check the accuracy of their ROI projections.

  • Managers are expected to meet often inaccurate ROI numbers.

  • Managers forget ROI's real purpose: making resource- allocation decisions.

    For a complete transcript of this roundtable discussion click here.