Roundtable: The Trouble With ROI

Something is wrong with this picture. In December 2002 CIO Insight surveyed
378 IT executives on their attitudes and practices regarding return on investment.
The headline of the results we published in March 2003 sum up our findings:
“Do You Have Any Faith In The Numbers?” Even though 60 percent of the respondents
said the pressure to demonstrate ROI had increased in the past 12 months, and
76 percent said the pressure to place a dollar value on intangible benefits
had increased, many IT executives found ROI to be a frustrating and difficult
exercise. Fully 70 percent conceded their metrics don’t adequately capture the
business value of their projects; indeed, half weren’t sure their companies
used the right approach. Many simply distrusted their own numbers: 46 percent
did not believe their methods were accurate, and 44 percent believed the results
of their ROI calculations were subjective.

We were intrigued to find CIOs so dissatisfied with a metric that they are under
increasing pressure to use. It led us to wonder just what’s causing so much
discontent—indeed, whether the concept of ROI needs to reconsidered, or
even abandoned in favor of a different way of looking at the value IT brings
to companies. And so CIO Insight Executive Editor Allan Alter, on August 11,
convened a discussion at the magazine’s editorial offices to discuss the problems
and limitations of ROI. Participants included CIOs, financial executives and
business management experts, including an IT executive who refuses to use ROI
at all. Are there viable alternatives?

The following is an edited transcript; For the full transcript please click here.

CIO Insight: Is the often elusive quest for ROI a symptom of a bigger problem,
like a lack of trust between the business side and the IT side?

Barkley: One of the former presidents of our company was always fearful
of “the system that ate Chicago.” So yes, you’re right. There’s a fair amount
of mistrust of the financial analysis capabilities of information technology
people. And there’s a fair mistrust of the technical competence of the financial

Murphy: The trust issue goes beyond IT and finance. Our finance group
requires ROIs for any capital expenditure beyond $150,000, and that includes
building a new ship. Once I, as CIO, have established that I have a methodology
and my team has process and discipline for doing these things effectively, however,
that changes the conversation. That’s where the trust comes. But that’s no different
from any other business unit.

Hirji: I think IT professionals probably brought this upon themselves
in some way. When I was a consultant, it was not uncommon to have projects between
$200 million to $500 million in size, most of which failed. They got three-quarters
of the way through implementation, but the systems didn’t get turned on. So
it was easy to calculate ROI on those—zero.

What’s the problem here? Poor leadership?

Riazi: They say the life of a CIO is three years, because the first year
is infrastructure, the second year is begging for money to do ERP and the third
year is when you actually do it. And by the third year, you have the whole organization
ready to shoot you. You learn as a CIO how to separate those who hate you from
those who are undecided. How do you get ROI? I think we start looking at our
failures, because our success has been abysmal. Our credibility is really shot.
And that’s why I refuse to go to see my CFO with an ROI case; he’s going to
laugh at me and kick me out of his office. [Laughter]

So how does any CIO survive?

Riazi: I don’t use ROI. I don’t think CIOs should use ROI because the
“R” isn’t in my department; the “I” is in my department. The R is in the business,
over which I have absolutely no control. So why should I use it?

Cameron: I disagree. The bottom line is not that ROI fails or that the
projects fail or that maybe there’s a bad odor of IT history. It’s that we in
IT have tried to do these things alone. The cultural change is for us to become
part of the business so that we relate to the rest of the firm like a manufacturing
plant general manager does.

Fletcher: You need to get over that sort of silo thing. The R and the
I all end up in shareholder value.

Gardner: I don’t think that we can escape this question of how we measure
the value of IT, because management will always have a problem with IT as long
as you can’t measure its value.

Cameron: Is there a value for IT? I would argue there isn’t. There’s
a value for business execution or delivery or value delivery with IT as a component.

Gardner: Which is fine. You’ve got to look at what IT is doing to contribute
to the overall value of the company. I think there’s also an issue around precision.
I remember seeing a vendor ROI analysis that came up with a number like 3,287
percent for some system. That’s just not the way to think about these things.
And you need to look at risk and return together, and risk is completely left
out of the equation. For those things that you can’t quantify and that are very
uncertain and ambiguous, one can at least make an estimate of the risk. Is it
something that could affect the entire success of the project, or will it have
a negligible effect? You can at least do that kind of thing.

Kalafut: We know there is no magical power in the technology itself.
Erik Brynjolfsson and his colleagues at MIT have studied companies’ investments
in IT and found that they’re almost invariably accompanied by much larger investments
in intangibles such as training and by the empowerment of employees as well.
All of it actually comes down to how you empower people to use those technologies
rather than the technologies themselves.

Atti, are vendors part of the problem?

Riazi: Yes. They come in, they sell you a concept that just looks terrific
and wonderful, and they promise the ROI without sharing the issues related to
change in the organization with the CFOs and the CEOs. We are discussing an
illogical issue in a very logical way. We are talking about people, and when
you implement technology, you change the way they work. ROI fails because we
forget these are people who are going to worry about their jobs and their training
and experience, and wonder what’s going to happen to me. Hundreds of people
have lost jobs, but that’s considered irrelevant. Between the vendors and us
CIOs, there is a responsibility to look at what happens when technology changes
the way an organization works. But the vendors have underestimated the importance
of that. They say you put this ERP in and it’s going to reduce head count and
make you more efficient and better. You put CRM in and you’re going to automatically
increase customers. It’s not real.

Schwartz: There are really two aspects to how vendors use ROI. One, they’re
trying to make a compelling value proposition to get the CIO’s attention. Two,
they’re trying to help their clients and customers create an internal justification
to build a business case. Some vendors are going to do this very well, and others
are not going to do this very well at all. They’re learning as well.

Fletcher: It seems to me that it’s very difficult to expect the vendor
to have an understanding of what is going to drive the return on investment.
For one thing, the discount rate that is used on the return on investment must
reflect the capital costs of the company, not of the vendor. Second, I don’t
know how a vendor could possibly understand all of those soft, cultural costs
that Atti was talking about. It’s hard enough for internal corporate management
to assess those costs.

Schwartz: Vendors will never understand their customers’ business the
way their customers do. You know your businesses inside and out. But what vendors
can do is bring a very valuable outside perspective and perhaps some other experience,
some benchmarks, some research on what other companies have achieved.

Riazi: I want a vendor that understands my business and understands my
problem. It’s their job to understand it if they’re going to sell to me. I rarely
get a vendor in my office that says, “I understand you’re facing this problem
in the advertising business, and I believe this could address this part of the
problem. It’s not going to be easy to put it in. I can promise you it will be
hard. And the users may not want it because it’s going to change the way they
work. But I think it’s the right solution.” That is so much more genuine than
a person who just comes to me and says this is going to give me a competitive
edge. Partnership is crucial to the success of ROI.

Murphy: I cannot imagine such a scenario where we would have a project
go forward at Royal Caribbean where I was developing the ROI. It isn’t done.
The business unit has to be accountable for the return on investment, except
for a few real core infrastructure-types of initiatives, and even then I can
show the value to the business.

Barkley: We require that the business case on every project over $1 million
comes to my desk. Before it gets to my desk, it’s signed off by the business
CFO. Even after he signs it, I’ll call him up and ask, did you understand what
you signed? Do you understand how you’re going to find a savings? Because I’m
not the guy who’s going to look for them, you are. We require that.

Okay, so despite problems around the concept of ROI, it seems like ROI is
still something people need to measure. Is that right?

Barkley: Maybe we’re overly focused on ROI. Here’s why. At our company
we suddenly realized, in looking at our budget process over the past two years,
that we spend 85 percent of our time on 30 percent of the budget. Projects are
important, capital investment is important, but 68 percent of our money is to
keep the lights on and the business running. We hadn’t done a good job at budgeting
and analyzing that. We had to figure out a way to do a better job of budgeting
“keeping the lights on” spending, and aligning those systems with the strategic
intentions of the company. So we’ve actually changed our budget process where
we do the lights on budget first. Then late in the budget process, we get the
project budget from the businesses and go through a strategic alignment value
analysis where we measure, among other things, internal rate of return, business
value risk and dependency.

Fletcher: A couple of thoughts. ROI is a good tool, I think, as long
as it is used consistently throughout the company for all kinds of different
decisions, whether it’s marketing, HR or anything else. You can’t just say IT
goes the ROI way but marketing does its own spiel. That doesn’t work. Second,
I think ROI is a very useful tool for allocating funds if you have a lot of
demand for those funds, and you have to make a judgment call between one versus
another. If you don’t have that kind of situation and you have all the funds
that you need in the world, I think free cash flow is a much better tool. It
too, though, is subject to lack of specificity or subjectivity because, let’s
face it, we’re all human and we all make projections from right now. If you
ask people to make a projection now about revenue growth, their projection would
be very different from anybody’s projections three years ago. I think the most
important thing is always the strategic driver. Why are we making this investment?
If I have only a $100 million and I have to decide whether I’m going to go with
Atti’s project or I’m going to go with a marketing guy’s project, I’ve got to
make some kind of an allocation decision. But if I don’t have that challenge,
as long as it’s free cash flow positive, let’s go do them both.

Bobby, is looking at IT investments the way investors look at portfolios
a helpful way to look at and think about ROI?

Cameron: Let me make a couple of comments before I answer that. One is
that it really is about the process. If the process doesn’t involve business
and IT equally, nothing works. Second, we’ve focused on ROI as if it was the
be all and end all, but not everything can be justified. Neither mandatory nor
biggest-customer-tells-me-I’ve-got-to investments are justified based on ROI.
Return on investment is really calculated for a portion of the investment stream
that is justified based on changes to operations. There are firms committed
to ROI that also justify projects based on softer metrics. EMC Corp., for instance,
uses soft metrics for things that might improve customer satisfaction, but would
be very difficult to prove an ROI.

That lays the ground for what I’m about to say, which is that most firms want
to invest to drive competitive advantage and improve stakeholder value. Frequently,
firms do their darndest to understand what the strategies are, and what they
need in order to accomplish these goals. The problem is there’s usually no linkage
between that planning and investments in the prioritization process. What I’ve
seen emerging is, in fact, a portfolio mechanism, meaning we look at everything
as a group of investments and we compare them to each other, and that the process
attempts to adjust priorities to strategic needs. But there is no correlation
in the data we’ve picked up between having a portfolio and success at prioritization.

So what are the two or three things that companies need to do first and foremost?

Cameron: The best firms start by committing capital investment dollars
in specific buckets. For instance, if I have $100 million to invest in capital
projects, I say 60 percent is going to customer value, 30 to value creation
(manufacturing or back-office systems) and 10 to enabling. By breaking them
into buckets, you start to clean up the conversation. Then, those buckets have
specific business goals they’re trying to achieve. Improving employee productivity
might be a good one in the value creation. Then the third step is to prioritize
projects based on their impact on those objectives, some of which are ROI and
some of which are soft.

Denise, you were a CFO. Would this approach pass muster on the business side?

Fletcher: Well, I start off with shareholder value. That is the driver,
nothing else is. Then the next step is, what drives shareholder value? The only
thing that studies have shown that actually, truly correlates to shareholder
value is free cash flow.

In my view, the key is not bucketing or anything like that. The key is to start
your process with a strategy. Where are you taking the company? Once you’ve
identified where you’re taking the company, the next step is to identify the
strategic drivers that are actually going to shift the company to where you
want it to go. Then you look at the strategic drivers and you ask, what IT,
what infrastructure do I need to make the strategic driver happy? And if I can’t
swing that, I obviously have to get rid of some of the things that I’ve been
doing until now. As I see it, the benefit of going through a process like this,
where you identify the strategy first, is that IT flows in as an enabler of
the strategy, not as somebody trying to implement a specific thing.

Hirji: I think the cardinal rule has to be to keep it simple. Anyone
can take something and make it overly complex. It looks very precise, and tons
of man-hours get chewed up doing it. Everyone feels better, but the reality
is you’re no further ahead. So the real genius is making decision-making simple.

Let me walk you through how we do it. We have specifically identified what our
strategy imperatives are. We have very clear goals and we can align all the
projects against them. If it doesn’t meet one of the goals, it’s off the table.
Next, every week the business and technology folks get together and run through
the prioritization, and they list it one through ten in terms of what they want.
We do that weekly because our projects are small and the marketplace changes;
what was important last week may not be important this week. Before any project
gets on that list, it has had a business case developed, which has a number
of things in it that say this is why it’s important. Sometimes it’s ROI; sometimes
it’s a bunch of things. Whether it’s ROI or not is irrelevant. The important
point is that that process gets both the IT folks and the business folks to
say we jointly believe this should be on the list, we think it should be number
X out of 50 on the list. It’s irrelevant what the number is. What’s more relevant
is getting the alignment.

Fletcher: The thing about ROI is this: The input is critical to the output.
Let me explain. I was CFO of MasterCard until a little while ago. One of the
major investments that we decided to make was in brand. Investing further in
brand meant that we were going to get more response from customers, and to deal
with that we needed to add storage capacity. If I look at my investment in brand
and include in that my storage capacity, it will give a very positive return.
If I look at storage separately, it is not likely to give me a very high return.
So I think the most critical thing is investment with the overall strategy of
the company. I think ROI is only a tool, and the tool is only as good as the
input that goes into it and as good as the boundaries that are defined around

Atti, does this begin to address your concerns about bringing back the human
element when looking at value?

Riazi: I’m not certain. I’ve heard a lot of great ideas here. But we
always fall back on our comfort zone, and I think we are falling back to a logical
approach to a problem which I find to be emotional. The best way to make ROI
work is to look at an organization organically. You have to look at it strategically,
look at it organically, and acknowledge IT will help, but ROI shouldn’t drive
it. The reason ROI has failed miserably is because it’s been sold as a driver,
and that’s when it becomes very difficult for it to deliver. I think we have
sold technology to the business in isolation. Here it is, you put it in and
it’s going to be wonderful. I call it the diet pill: You take it, you lose 50
pounds and you don’t have to worry about anything else.

Let me share one story about my 8-year-old nephew. I went to buy him a Halloween
costume; he wanted to be Superman, so I bought him a Superman costume. I took
the costume home and I looked in the back and it said: “This does not enable
you to fly.” [Laughter] That’s an important lesson for us. The message
out there is all this wonderful technology stuff is going to make you fly. It
doesn’t, and it won’t. Somehow we need to come to terms with that.

CIO Insight Staff
CIO Insight Staff
CIO Insight offers thought leadership and best practices in the IT security and management industry while providing expert recommendations on software solutions for IT leaders. It is the trusted resource for security professionals who need network monitoring technology and solutions to maintain regulatory compliance for their teams and organizations.

Get the Free Newsletter!

Subscribe to Daily Tech Insider for top news, trends, and analysis.

Latest Articles