Participants:
- Joseph Barkley, Vice President and CFO for Global Information Technology,
American International Group Inc. (AIG) - Bobby Cameron, Principal, Forrester Research Inc.
- Denise K. Fletcher , Managing Director, FA Group
- Christopher Gardner, Partner and Cofounder, iValue
- Asiff Hirji, Executive Vice President, CIO, Ameritrade Holding Corp.
- Pamela Cohen Kalafut, Director, Intangibles Practice, Cap Gemini Ernst &
Young - Thomas H. Murphy, CIO, Royal Caribbean Cruises Ltd.
- Atefeh(Atti) Riazi, Senior Partner and CIO, Ogilvy & Mather Worldwide
- Julie A. Schwartz, Vice President of Research, Information Technology Services
Marketing Association
CIO Insight: Let me start with the first question: What is ROI? What does
it mean? Does everyone agree on what it means? Atti, could you start us off
on that? How do people usually define ROI?
Atefeh(Atti) Riazi: It depends who defines it. Mostly, the CFOs define
what return on investment is, and it’s defined from the financial terms. I think
a lot of what we do in IT has changed and has evolved. For example, you cannot
really use ROI in the way we know it in knowledge management. There are so many
areas where you cannot apply the financial point of view to it because the return
is not necessarily financial.
So who’s defining it, why is it being defined? I personally, and I’ve shared
this with you, I would say most CIOs are so fed up with the vendors who come
in and sell this virtual vaporware with this great return on investment, which
is nonexistent. You’re kind of pushed to get to the CFO’s office and say here,
yes, there is an ROI. And I refuse to do it.
CIO Insight: We’ll get back to talking about the vendors later. Joe, you’ve
been a finance executive, working with IT organizations for many years. Is there
a clear definition of ROI?
Joseph Barkley: The answer is yes and no. ROI is an accounting indicator
that CFOs are comfortable with, because they know how to define return on investment,
there are several mathematical formulas to get to that, and it’s a number.
If you look at the value of IT and tie it to a financial number, that analysis
is generally doomed to frustration and failure. ROI is merely one indicator
that you can use to rank the project. One of the problems we have with ROI is
we can’t find it after we finish the project, because the project and the effect
are melded into the rest of the budget and the operation. We haven’t figured
out how to go back and find it, verify all of the savings supposedly or all
of the effect. Unless you’ve got a discrete business, by the way; then you can
do it.
CIO Insight: But is there a specific definition of ROI within the finance?
Denise K. Fletcher: Yes, of course there is one. However, just like every
other kind of modeling, the input is critical to the output. So if you start
off your analysis [by] defining a project very narrowly, it is not likely to
be representative of the investment.
Let me explain what I’m saying. If you look at a project, for example, [and
say] you need to add storage capacity, and you say it’s going to cost X number
of dollars [and], I’m going to get Y return on this project, to me the weakness
is that you have to step back to be able to get to the right place. You have
to step back and say: How does this fit into the strategy of the company? You
can’t separate an IT project any more than you can separate a marketing project
from the overall strategy. If you step back and you look at the overall strategy,
I think that you will find that just about every project has a purpose and a
return.
To be very specific, I was CFO of MasterCard until a little while ago. One of
the major investments that we decided to make was to invest in brand. Investing
further in brand meant that we were obviously going to get a lot more response
from customers. But to be able to deal with that, we needed to add a lot of
storage capacity. Now, I can look at my investment in brand and include in that
investment my storage capacity, which is going to give a very positive return,
or I can look at it separately which is not likely to give me a very high return.
So I think the most critical thing is to go back to a word that [CIO Insight
Editor-in-Chief] Ellen Pearlman used earlier: it’s the alignment, the alignment
of the investment with the overall strategy of the company. I think ROI is only
a tool, and the tool is only as good as the input that goes into it and as good
as the boundaries that are defined around it.
Bobby Cameron: May I make a comment?
CIO Insight: Go ahead, Bobby.
Cameron: I think the series of comments point out two really critical
factors. The last one I think is the most important, and that is there is no
such thing as an IT investment, with the sole exception of some pure infrastructure
cost-saving kinds of things. The justification or the rationalization has to
be a complete view, meaning the business and the IT parts. There is no such
thing as a standalone IT business investment. The second is that the mechanism,
whatever it is, that goes to [the] justification of an investment has to be
one that sits with the overall business operation, meaning if ROI or MPV or
EVA or whatever is used for business investments, that should be what’s used
for IT. The flip side, which is frequently the case, is that IT is held to criteria
that the rest of the business isn’t held to. So, if IT is asked to do ROI and
the rest of the firm isn’t, then CIO’s feel like they’ve run against the wall.
CIO Insight: I’d like to get back to that, Bobby, but let me get Tom Murphy
on the phone. Tom, you told me that ROI has to be very closely connected to
alignment as well. Could you comment?
Thomas H. Murphy: Well, I agree with Denise and Bobby, and our approach at Royal
Caribbean is [that] we have a very clear business strategy that I’ve actually
coauthored, and we have then aligned the IT strategy to support all of those
business elements. We’ve laid out a $200 million investment called Jumpstart
that has been approved at the board level with no ROI.
CIO Insight: What is Jumpstart?
Murphy: Jumpstart is our core business systems rewrite over the next
three years. There are elements of employee, guest and supply chain to all of
those, and we have component projects that make up that investment, and that
is aligned exactly with the strategy that the business has put forward. In fact,
the board has accepted that the business cannot achieve their strategies if
we do not execute these IT projects that are really business projects.
But each of those projects, as they come up, are prioritized by the business
and justified by the business for their value. In some cases those [justifications]
are very ROI-driven, but in most cases they’re strategic or fit into this jumpstart
model. They still have to go through an approval process, but ROI is not the
sole measure. There are many measures of that and, again, it aligns very neatly
with the strategy that’s been laid out for the next three to five years by the
business.
CIO Insight: I’m hearing that everyone’s emphasizing the importance of alignment.
Let’s look at a numbers themselves. You were telling me about the ROI numbers
that themselves are often subjective and even illusory. Could you talk more
about that?
Asiff Hirji: I think that there’s a vested interest in pushing ROI from
many areas, because it gives decisions which are by their nature difficult to
substantiate, and difficult to get a group of people to align on, the look and
feel of specificity without, frankly, very much specificity in many cases. If
you look at the way ROI typically came across, anytime you have a set of inputs-this
is back to Denise’s point-that are subjective in their nature, the measurement
is, even though it’s mathematical, still a subjective measurement. So if you
have any sort of terminal value, if you have any sort of any long-term profit
stream that’s baked into there, you will create something that has this sort
of taste and smell of precision without any of the actual benefit of precision.
Anyone [who] comes to the table and says, yeah, this is a big number, it’s bigger
than that number, definitely let’s go for this, [is] deluding themselves.
[The] objective here is how to make better resource allocation decisions. Whatever
works for your firm, you have to find a way of matching the technology decision-making
to that process. If your firm throws darts at a board, great, keep doing that
for technology because that’s part of how you make the technology decision-making
in line with the decision-making of the business. And I think, again to the
point that Tom was making, that better companies will have a very tight alignment.
There’s no such thing as a technology prioritization. It’s business prioritization
across various business lines, that flows to a set of projects, which flows
to a set of initiatives. The way we do it at Ameritrade, every six months we
revisit the macro level allocation across all the businesses. That’s not just
technology, but marketing, HR, the whole bit.
CIO Insight: Chris, do you accept that it’s impossible to have objective
ROI measures?
Christopher Gardner: No, having written a book on the topic, we have
strong opinions about this, and everything that’s been said today is absolutely
what we’ve encountered. It’s a very hard problem to solve. There are problems
of isolation, alignment, the economics of companies vary, so the ROI definition
varies from company to company. In fact, ROI in my mind is more of a proxy for
a broader issue which is what is the value of IT, and how you measure that can
be done in quite a variety of ways, some of which follow financial guidelines,
[and] some of which are internal indices that are more meant to be something
that correlates with value.
There’s a bunch of additional problems with ROI. One problem is that the inputs
tend to get done poor, that there isn’t enough time assessing demand in a rigorous
way. There isn’t enough time looking at alternative designs and assessing those.
The economics of the business are not assessed and analyzed so that the IT system’s
effects on the economics of that business are clear.
I think also, aside from kind of the hard work in getting the inputs right,
the numbers can be dangerous. If they’re not done properly, usually they’re
wrong, and then people get held accountable to numbers that are wrong. And so
you have this situation where, frankly, I think there’s an obligation to get
the numbers right if you’re going to hold people accountable, and the fact that
the numbers can be dangerous is another issue here. But the inputs are the most
critical thing to get by here.
Julie A. Schwartz: I’m listening to the conversation, and it strikes
me that there still might be too much emphasis on the numbers, the ROI tool
and calculation itself, and not enough on the process. Chris, I think you’re
absolutely right about the fact that sometimes the numbers can be wrong, and
as part of the process what’s missing is this test for reasonableness. I’m on
the panel representing the vendor point of view, and one of the things that
the vendors can be very helpful with, by working with their clients and their
customers, is getting benchmark data together so they can build a database to
help companies who are struggling with this problem to look at what is possible,
what is reasonable. The other part of the process that’s always missing is the
postimplementation evaluation, going back and saying, okay, how were our numbers?
Did we reach what we were going to achieve? What we found is that [in] very
few, you know, maybe 20, 30 [percent] of the projects, [do] people go back and
say, well, what did we do with ROI based on what our inputs were and what we
got out? The problem is, we’re not learning. There’s no learning from the experience,
so we’re not improving the process in any way, shape or form.
Gardner: It’s actually borne out in your CIO Insight survey. ROI analysis
is done at the very beginning of a project, but after that, there’s very little
effort to follow through.
Cameron: Let me give you just another data point, some data we at Forester
gathered at the start of this year. The numbers bear out very clearly what Julie
just quoted. We saw on average only 37 percent of projects were measured postproduction,
but the interesting thing is that when we took that sample [of] senior IT decision-makers
at 174 $1 billion plus North American firms, and broke it into quartiles-in
other words, those who did the most down to those who did the least measuring
postproduction-what we found is that the top quartile measure 86 percent of
their projects postproduction, 45 percent the second quarter, third quarter
13 and the bottom 1 percent. So my point is that those who learn how to measure
and learn how to do it well do probably as much as they possibly can.
CIO Insight: I want to go back to Atti. You were talking before about vendors.
We heard from Julie how they can help and what they can do. But do you feel
that, in reality, they’re helping as much as they could or that they’re actually
part of the problem?
Riazi: I think some of the way they’ve sold IT has been the ROI concept,
and I think they are part of the problem in many ways because they never talk
about the real issues. They come in, they sell you a concept that just looks
terrific and wonderful, and they promise the ROI without sharing the issues
related to change in the organization with the CFOs. The CIO becomes the therapist
or the catalyst to make that change happen.
We are here discussing a very irrational, illogical issue in a very logical
way. And that’s the disconnect I feel, that’s the lack of alignment. We are
talking about people, and when you implement technology, you change the way
they work. And that is very irrational. The reason ROI fails is because we forget
these are people who are going to worry about their jobs and their training
and experience, and what’s going to happen to me. I haven’t seen any research
data or statistics on what happens there. Because the analysis and the statistics,
all of them, are really a logical view: The process was very long, we collapsed
it; the hierarchy was high and we collapsed it. Meanwhile, hundreds of people
lost jobs, but that’s irrelevant; that’s not a logical argument.
So I think between the vendors and us CIOs there is a responsibility to look
at the failure, and the failure is not a logical discussion. It is a discussion
of what happens in an organization when that technology changes the way that
organization works, and it’s a lot of soft issues that happen. The biggest thing
is what’s going to happen to me, and that is going to stop your ROI. It’s going
to stop the return because [people] are not going to want to change. But the
vendors have underestimated the importance of that. They too have approached
[technology changes] in a logical way. You put this ERP and it’s going to be
great and it’s going to be less head count and more efficient and better-you
put the CRM in and you’re going to automatically increase customers. And it’s
not real.
CIO Insight: Julie, I’d like you to respond, and then, Pam, this would be
a wonderful moment to get you in to the conversation.
Schwartz: I think you make some very good points, but what vendors have
learned is that with very few exceptions they are not able to get the kind of
partnership relationship with their customers that they would like, so that
they could walk hand-in-hand with the customers through the process you describe
of really understanding the change management that goes on in the organizations
where their solutions are implemented. In many cases we don’t know how. [Vendors]
are engineers, and they’re very analytical, and they don’t know how to deal
with that kind of ambiguity.
But I think it’s important to keep in mind that there are really two aspects
of how vendors use the ROI. One is that they’re trying to make a compelling
value proposition to get the CIO’s attention. You know, you sit there, every
week you get hundreds and hundreds of unsolicited e-mails and phone calls. And
[vendors] need to find a way to get in so that they can tell you about their
solutions. So one thing they are doing is trying to come up with the most compelling
value proposition, and that might very well include ROI that looks like smoke
and mirrors or vaporware to you.
The other reason they need to have ROI models is to help their clients and customers
with an internal justification to build a business case. Some vendors are going
to do this very well, and others are not going to do this very well at all.
They’re learning as well. But the most important thing is for the customers
to really open [up] and say [to the vendors] we’ll work with you so that we
can collect the data, we can build the models, we can look at the assumptions,
and do this in a consistent way so that you can do this with other customers,
so that we can build a database so that we can track what’s reasonable, so that
we can start learning from our experience going forward so that we can make
this a better tool. It’s more of a process than an output.
Pamela Cohen Kalafut: I think that all makes sense. One of the things
we get asked about the most is how this measure or others are going to help
businesses get some sort of competitive advantage, which, as we all know, is
almost impossible to sustain in the present economy, except through measures
that might not always be recorded by businesses.
Fletcher: If I could just make a comment. It seems to me, if you start
off with a premise that ROI is part of alignment and part of a corporate strategy,
then it’s very difficult to expect the vendor to even have an understanding
of what is going to drive the return on investment. For one thing, the discount
rate that is used on the return on investment must reflect the capital costs
of that company, not of the vendor. That’s the first weakness. Second, I don’t
know how a vendor could possibly understand all of those soft costs that Atti
was talking about, the cultural costs. They’re hard enough for internal corporate
management to assess, let alone a vendor. I think the weakness, as I hear this
conversation, of the approach is that when an outsider is looking at [ROI] they’re
looking at a little box, but when you look at an ROI calculation to do it right,
you have to look, as I said earlier, to the whole [set of costs], which really
drives the responsibility of the ROI to internal management and not to the vendor.
Schwartz: The vendor is there to help in any way that they can, but they
cannot do [the ROI calculations] for the customer. They can provide data: Have
you thought of XYZ? Have you thought of ABC? Other customers we have worked
with have done it this way. But they can’t do it. In fact, we have done research
with customers and asked who do you want to do your ROI calculations, and the
majority say that they’re going to do it themselves, and a few are saying they’re
going to do it with vendors helping them. But for the most part, they’re going
to do it themselves.
CIO Insight: I have a question. Is ROI, in general, a symptom of a bigger
kind of problem: a lack of trust between the business side and the IT side,
and [of] being able to prove that a technology investment is going to return
something to the company? If there was better communication, perhaps there wouldn’t
be so much focus on that. Is it a reflection of that, or [of] something else
going on in the enterprise?
Barkley: I’d like to get into that first. To quote one of our former
presidents of our company, he’s always fearful of "the system that ate
Chicago," because systems tend to grow. So you’re right, there’s a fair
mistrust. There’s a fair mistrust of the financial analysis capabilities of
information technology people. Don’t take that the wrong way. And there’s a
fair mistrust of the technical competence of the financial people, which is
one of the reasons there’s now people like me in our information technology
[organization]. Because at the end of the day, the CIO and the CFO have to agree
on the strategy, agree on the costs, and agree on what we’re going to deliver.
But there’s a fair amount of mistrust.
Murphy: I think the trust issue goes beyond IT and finance. Our finance
group requires ROIs for any capital expenditure beyond $150,000. That includes
building a new ship, which is a little bit like putting in a custom application.
It’s hard to nail it down. It’s a physical asset, but it has a tendency to morph
and grow and become more expensive as people have new, clever ideas, and a lot
of it comes down to effective process and project management. I think the trust
part comes in that once I, as CIO, have established that I have a methodology,
and my team has process and discipline, for doing these things effectively,
that changes the conversation. But I don’t think that’s any different than any
other business unit that has significant capital requirements to do their business.
CIO Insight: Asiff, you had a comment?
Hirji: I think the sentiment in that question [about lack of trust] is
right. I think IT professionals probably brought this upon themselves in some
way. There was a study about how many major systems initiatives actually deliver,
and the vast majority of systems fail. When I was a consultant, it was not uncommon
to have projects of $200 million to half a billion dollars in size, the vast
majority of which failed. Forget didn’t deliver ROI, didn’t get turned on. [Projects]
got three-quarters of the way through implementation, and didn’t get turned
on. So it was easy to calculate ROI on that-zero. I think those days, thankfully,
are dead, and that we collectively have learned a lot. But if you have a process
by which your business allocates resources, checks progress on the projects,
makes effective decisions about whether they’re proceeding or not proceeding
properly, reallocates and delivers shareholder value or earnings per share or
whatever it is you’re aiming for, it doesn’t matter if you use ROI or not, it
doesn’t matter what tool you use to do that, so long as the process is working
and you as the management team are working effectively with it.
I think ROI is great for vendors, and I don’t blame them for using that. It’s
an effective mechanism. But if you’re going to use it, you better use it in
context of the way your organization makes decisions and in alignment with the
way your organization makes decisions. It’s like code: It’s only as good as
the inputs. You can write the best code in the world, if you feed a bad input,
you’ll get bad output.
So, there’s too much emphasis on ROI as a measurement mechanism, and there’s
not enough on what’s it about. Go back to that study that I think our friend
from Forrester, Bobby Cameron, was quoting, he said top-tier companies do track
ROI and they do track return on investment and so on. But I think CIO Insight’s
study said that people have different definitions of ROI. So my takeaway from
that is companies that set some goals in advance, measure the project’s process,
then measure against those goals what the project actually did, they learn and
they get better. It doesn’t matter whether you use ROI as part of that or something
else as long as you are setting some goals, doing the project, checking it against
the goals afterwards.
And in our experience at Ameritrade, the more you shorten the lifecycle of that,
the better off you are. We will not do anything that is longer then 12 weeks.
If we can’t do it in 12 weeks, it’s dead. It doesn’t get off the ground. Why?
Because as bad as we all are at estimating, if we’re estimating from six to
nine months out versus six to nine weeks out, we’ll take the risk on six to
nine weeks. It’s just human nature.
Barkley: Let me just pick up. We worry about delivery. Once we approve
the project, ROI [is] one of the indicators, but we’re more interested in the
alignment and technology for our strategy. We’re interested in the delivery
of the project in the budget, and don’t be late. We worry about those two things
more than anything else. The other thing we started to do this year is every
time we develop or design a new application or a new project, we put a metric
in there so we know if it’s working. Let me give you an example. If you build
an application that’s going to do policy issuance and it’s supposed to be able
to issue a million policies a month and you run it and it issues a policy, [people]
say it works. Well, it doesn’t work. It only issued one policy. I want to see
it issue a million policies. Then I know it works. We have not in the past had
operational metrics. When we install them, we started to improve our delivery
in our continued operations, and the value, by the way.
Fletcher: Well, a couple of thoughts. I think that there was a comment
made earlier about specificity and what goes into the equation is kind of a
little bit random. I’d like to answer that. In my view, ROI is a good tool as
long as it is used consistently throughout the company for all kinds of different
decisions, whether it’s marketing, HR or anything else. You can’t just say IT
goes the ROI way but marketing does its own spiel. That doesn’t work. Second,
I think ROI is a very useful tool for allocating funds if you have a lot of
demand for those funds, and you have to make a judgment call between one [demand]
versus another. If you don’t have that kind of situation and you have all the
funds that you need in the world, I think free cash flow is a much better tool.
It too, though, is subject to some of what we call lack of specificity or subjectivity
because, let’s face it, we’re all human, and we’re all going to do projections,
and we all make the projections from the time in history of which we are living
right now. If you ask people to make a projection now about revenue growth,
they’re going to give you one kind of projection, which would be very different
from anybody’s projections three years ago. We never have the ability to forecast
cycles. It’s always a straight line from whatever point we’re in, or a hockey
stick.
Where I’m going [with that] is I think the most important thing is always a
strategic driver. Why are we making this investment? Is it because we want to
increase knowledge? Okay, that’s great, we want to have certain kinds of efficiency.
Let’s not kid ourselves that we need to have a return on investment. But if
I have only $100 million dollars and I have to decide whether I’m going to go
with Atti’s project or I’m going to go with a marketing guy’s project, and they
each want $80 million, I can’t give them $160 million if I’ve only got $100
million. So I’ve got to make some kind of an allocation decision. But if I don’t
have that challenge, as long as it’s free cash flow positive, let’s go do them
both.
CIO Insight: Atti, I was fascinated by your comment on culture, essentially
the soft issues. I wonder if you could elaborate a little bit about this. I’m
hearing about credibility problems, I’m hearing different approaches to who
has leadership here. We also hear a lot about rogue IT projects. I’m just wondering
what this is a symptom of. Is this a leadership problem? How do we fix this?
Riazi: I wish I knew the answer. I’ve struggled with this issue the last
20 years being in IT and being a CIO. I think there’s one reason they say the
life of a CIO is three years, because the first year is infrastructure, the
second year is begging for money to do ERP applications, and the third year
is when you actually do it. And by the third year, you have the whole organization
ready to shoot you. You learn two things as a CIO [to be] able to separate those
who hate you from those who are undecided. (laughter)
This is a very complex issue, and I think, and I’m hoping through panels like
this and the research companies [to] start going to the forbidden zone. We all
hate to do [it]. We’re all analytical IT, finance people, and I think we’ve
lost focus on the real issues that I have faced in every project I have rolled
out, every large project. I was training some of the finance people, I was in
the room where they were being trained on a wonderful application, and three
of them sat back and said but that’s my job. I was showing them how it was going
to be automated, and the light bulb went off, that’s my job. Well, that was
the job of another 50 percent in the room, but the light bulb hadn’t gone off
yet.
How do you get ROI? You get it through people. I mean, the biggest cost is people,
so that’s the first way you’re going to go. Even if you don’t let people go,
you’re going to make them more efficient, they’re going to have to learn something
new and there is the evaluation on that. I think we [have to] start looking
at our failures, because our success has been abysmal. Our credibility is really
shot. And that’s why I said earlier I refuse to go to see my CFO with an ROI
case because he’s going to laugh at me and is going to kick me out of his office.
CIO Insight: So how do you survive? How does any CIO survive?
Riazi: I don’t use ROI. I don’t think CIOs should use ROI because the
"R" isn’t in my department; the "I" is in my department.
The "R" is in the business which I have absolutely no control. So
why should I use it?
Cameron: I disagree. I think you yourself pointed out an example earlier
where the expense sits on both sides, where the user was sitting down and recognizing
that’s my job, the investment really is from them, too. They have to put time
in; they have to change their process. Good technology has not nearly the impact
[of] good process change, with or without technology to support it. So the bottom
line, what I’m hearing you say is, not that ROI fails or that the projects fail
or there’s maybe a bad odor of IT history, it’s that we in IT have sat alone
and tried to do these things alone, and that really the cultural change is for
us to become part of the business so that we relate to the rest of the firm
like, let’s say, a manufacturing plant’s general manager does to the rest of
the firm.
CIO Insight: Is ROI politics? Is it possible to say that ROI is a technique
that business people use to put the onus and responsibility for IT failure on
IT people? You come up with an ROI, you prove it to us, and if doesn’t make
it, it’s your fault.
Murphy: I cannot imagine [such] a scenario where we would have a project
go forward at Royal Caribbean, where I was developing the ROI. It isn’t done.
The business unit has to be accountable for the return on investment, except
[for] a few real core infrastructure-types of initiatives, [and] even then I
can flip back and show them how the value is going to go to the business. Now,
you know, alignment is a big part, culture is a big part, trust is a big part.
I have a personal relationship with every businessperson in this organization,
and I am one of the businesspeople. I don’t think people view me as that IT
guy who doesn’t deliver on projects. I’m in a partnership with all of these
folks trying to drive the business to success. It’s a fascinating conversation,
and there’s a lot of angst out there obviously, but I think a lot of it is in
the relationships that you develop within the business and the trust and the
process that you put into place.
Barkley: We require a business case on every project over $1 million
that comes to my desk. Before it gets to my desk, it’s signed off by the business
CFO. Even after he signs it, I’ll call him up and say: [Do] you understand what
you signed? Do you understand how you’re going to find a savings? Because I’m
not the guy who’s going to look for them, you are. We require that. In our company,
the application development is decentralized; that is, we have CIOs in each
of the businesses that are matrixed to my boss, the global CIO and also to the
business executive, although we control all of the infrastructure centrally.
So the businesses have a big stake in the strategy, what we’re going to do,
and the returns as well. Again, ROI is merely one of the indicators.
Fletcher: I was very concerned by a comment that Atti made which was
that the return is at the unit, the "I" [in ROI] is with her. That
would concern me because I think that that, to me, would be a little bit of
a cultural disconnect between the IT organization and the rest of the organization.
The most important thing to remember is that the "R" and the "I"
are owned by the company.
So you need to get over the sort of silo thing, and you need to go, I think,
to the models that both Joe and Tom are talking about, where you have a really
strong alignment between the business units and IT, whether the IT is decentralized
or centralized. But I think there are some pitfalls thinking that the "R"
and the "I" are disconnected, because it all ends up in the shareholder
value, and the shareholder value is the company as a whole. You can lose sight
of that in the silos.
Gardner: I have to say I agree with everything that’s been said here.
I’m not doing it because I’m trying to form any consensus; I think everything
here is true. I like Atti’s comment about people problems. I mean some of these
soft benefits or soft costs, how do you get at those? There are ways to do that,
and we actually borrow techniques from the advertising industry to get at some
of these people issues, primary market research where it’s not a precise number
but it’s a good enough number, and you can start to see what the dynamics are
between some of the choices and some of the design alternatives.
CIO Insight: But how do you get at the people issues? I just want to make
sure I understand what those people issues are and how they relate to ROI.
Gardner: Let me just make a couple of other comments. One is that I don’t
think that we can escape this question of how we measure the value of IT, because
management will always have a problem with IT so long as you can’t measure its
value. If you can’t measure the value of IT, my sense is-[and] this gets back
to your point about is there trust between upper management and IT-if you can’t
measure the value of IT, how can you claim management, and the very top levels
of the organization, are held accountable for shareholder value?
Cameron: Is there a value for IT? I would argue there isn’t. There’s
a value for business execution or delivery or value delivery, with IT as a component.
Gardner: Which is fine. You’ve got to look at IT in the context of the
company, [at] what IT is doing to contribute to the overall value of the company.
Maybe it’s helping to create various competitive advantages. IT can be costly;
that can create a capital barrier. There are skills involved, some of the people
issues relate to IT skills, and that can give you can advantage. So, yeah, IT
has to look at the broader context that it sits in, and you’ve got to understand
how the company makes money so that the ROI [analysis] that you’re performing
ties to the way the company is actually making money. But I think there are
ways to get at some of these things. I think there’s also an issue around precision.
[With] ROI, I think, sometimes there’s a number that people are looking for
[with] several decimal points. I remember seeing a vendor ROI analysis that
came up with a number like 3,287 percent for some system, and that’s just not
the way to think about these things. Initially, you go in, you might do a back-of-the-envelope
type of calculation, and just see if it even makes sense to pursue further.
And then over time, as things proceed and get more serious and maybe more money
is involved, you make the effort to do the primary market research, to do the
careful estimate and adjust for the cost of capital and risk. You know, it’s
amazing to me that [in the] companies that we’ve served, there’s a lot of talk
and focus on ROI, the return part of the equation, but you need to look at risk
and return together. Risk is completely left out of the equation. For those
things that you can’t quantify and are very uncertain and ambiguous, one can
at least make an estimate of what’s the risk. Is it a code red? Is it something
that could affect the entire success of the project, or will it have a negligible
effect? You can at least do that kind of thing.
Kalafut: I just wanted to get at the question in terms of the human side
[of ROI], because the [human side of technology] is one of the things we talked
about in [our book] Invisible Advantage and as part of our value creation index.
We know there is no magical power in the technology itself. But in terms of
the human side of technology, we have the work processes that go into it which
are run by the people that actually have the power to create the strategies
around it, and [who] motivate others to actually use that [technology] and decide
that they’re not going to be replaced by those processes. Also, in terms of
work place organization, creating the jobs and responsibilities, the change
in how the training comes into play to actually absorb people with new talents
within the work place, has been one of our measures. Erik Brynjolfsson and his
colleagues at MIT have studied companies’ investments in IT, and found that
they’re almost invariably accompanied by much larger investments in intangibles,
such as training in the workplace, and also by empowerment of employees as well.
We go through a variety of examples, but all of it actually comes down to how
you empower people to use those technologies rather than the technologies themselves.
CIO Insight: I just want to step back now. We’ve talked now for close to
an hour about what the problems and the issues with ROI. The pressure is there
to come up with an ROI, but it’s still difficult to calculate. It seems to me
that this is still something that we need to do, at least as part of the larger
issue of value, particularly in the financial community. Does that sound right?
Barkley: It’s slightly different. Maybe you’re being overly focused on
ROI. Here’s why. We suddenly realized in looking at our budget process over
the last two years [that] we spend 85 percent of our time on 30 percent of the
budget. Projects are important, capital investment is important. But 68 percent
of our money to is to keep the lights on and the business running. We hadn’t
done a good job at budgeting and analyzing that. So we had to figure out a way
to do a better job at budgeting lights on, and aligning lights on with the strategic
intentions of the company. So we’ve actually changed our budget process where
we do the lights-on budget first, and then late in the budget process, after
the businesses have done their budget, we get the project budget from the businesses,
and we go through a strategic-alignment value analysis where we measure, among
other things, internal rate of return, business value risk and dependency. We
do that in the last few weeks of the budget [process]. While [the project budget]
is important and we need to have it to justify the projects, we’ve got a large
part of the business that we realize we hadn’t been paying attention to, and
we have to analyze that too.
Riazi: My problem is that I think we have sold technology to the business
in isolation. Here it is, you put it in, and it’s going to be wonderful. I call
it the diet pill: you take it, and you lose 50 pounds, and you don’t have to
worry about anything else. That’s the underlying problem.
Barkley: Do you know where that pill is? (laughter)
Riazi: I think technology has a great deal of ROI, it’s just our approach
to it, our mind-set [that is a problem]. The fact is [technology] enables you
to do so many different things. It enables you to streamline your operation,
if that’s how you approach it. [But] if you approach it that we’re going to
dump this and it’s going to be great and it’s going to give us the ROI, it’s
going to fail. So I’m not questioning the value of ROI. I believe it is [valuable]
and I believe we haven’t even scratched the surface. But there are other issues.
CIO Insight: What I’m hearing is we need to take ROI value out of isolation
for the rest of the business. We need to make this part of a process. We need
to use that process to build trust. And that the real purpose of ROI is to make
allocation decisions; that ultimately is really what it’s about. So, how do
we transform our narrow, isolated approaches to ROI to something that’s going
to really get at the question of value, and is really going to help us make
better decisions? Where do we start on that? Chris, can we learn something from
Wall Street and how Wall Street looks at valuation of companies?
Gardner: Absolutely. My feeling is the top of the company is measured
based on Wall Street style valuation models, and that should be reflected elsewhere
in the company. Those [valuation models for IT] can be built, they’re complicated
and they’re hard, but it can be done, and at least you have some assurance that
your analysis ties back to something that is meaningful from a top executive
standpoint.
CIO Insight: What would be the basic things needed to achieve that?
Gardner: Well, you need the free cash-flow approach that Denise was mentioning.
It’s ultimately about cash flow. We’ve talked about ROI here, but my feeling
is ROI is more of a general term that encompasses the concept of value for IT.
The free cash-flow approach, there’s strong evidence that shows it correlates
with share price better than earnings per share and a bunch of other things.
But this gets back to the inputs. If you’ve got a system that involves a lot
of customer adoption, you need to understand the customer’s response to the
design that’s been proposed. You’ve got to work out those costs in detail. You
have to be complete about your costing. You’ve got to make sure training is
in there and change management, [and the] inefficiencies, the delays and that
kind of thing. And some of these may not be known going into it, but you can
see, well, what if we’re off here by 50 percent or 10 percent or 100 percent?
Does it make any difference?
You can start to create something that you can use to manage the whole process
of building a system. What we’re talking about isn’t something you just use
at the beginning of a project to decide whether to invest in it or not; it’s
something of a tool that you use throughout the lifecycle of a project. So during
the development you would know what are the get-rights. You know, if you’ve
got a narrow market-opportunity window, schedules will be paramount. You can
show that and model that. Then [there’s] the operation of the system itself.
This is really something that should be an operational tool and tie back to
operational measures.
It’s hard, and there’s a lot of uncertainty here. The key is to always ask questions
and challenge it. Credibility is low right now, but a tool like this should
start to predict that some projects are not worth doing and some projects are
worth doing. IT should start to be able to show that the business might be recommending
projects that don’t make sense. This is a politically difficult kind of thing,
but you should start to see some projects being turned down. A test is: What
am I seeing coming out of this? Does it calibrate with reality? Am I getting
some responses that look halfway credible? Can someone show me the evidence,
make the case, you know, sometimes you have to put it on the line.
CIO Insight: Bobby, what do you think? Is looking at IT investments kind
of the way investors look at portfolios a helpful way to look and think about
ROI? Is that something we all need to be doing as part of changing and transforming
ROI?
Cameron: Let me make a couple of comments before I answer that directly.
One is that it really is about the process. No matter what you do, if the process
doesn’t involve everybody fully, meaning business and IT equally, and the justifications
don’t cover everything, nothing works. So just put that on the table and push
it aside. The second overall statement is that we’ve focused on ROI as if it
were the be all and end all, but not everything either merits [justification]
or can be justified, and so we need to make sure that we put some clarity there.
Most firms that identify minimum pain around prioritization isolate strategic
mandatory and operational investments into those categories where strategic
justification is different from mandatory, tax regulatory, biggest-customer-tells-me-I-got-to
[justifications], and neither of those is justified based on ROI.
ROI is really calculated for a portion of the investment stream that has to
do with things that are justified based on changes to operations. In other words,
I expect a business impact. And then there are firms who are committed to ROI
who also justify projects based on softer metrics, EMC being one, for instance,
where they use soft metrics for things that might improve customer satisfaction
but would be very difficult to prove an ROI.
So now that’s sort of laid the ground for what I’m about to say, which is that
most firms want to invest, as someone said earlier, to drive competitive advantage
and improve stakeholder value. Frequently firms do their darndest to understand
what the strategies are, what the directions are that they need to accomplish
these goals. The problem is [that] there’s no linkage between that planning
and thinking and investments usually in the prioritization process. What I’ve
seen emerging is, in fact, a portfolio mechanism, meaning we look at everything
as a group of investments and we compare them to each other, and the process
attempts to adjust the prioritization [of] strategic needs.
CIO Insight: So what are the first steps then that we need to do to change
and improve the way we do ROI?
Cameron: Well, ROI is best used when it’s a relative comparison tool.
It has no absolute characterization. What I mean by that is if I want to adjust
the direction of my spend to parallel what I’m trying to accomplish, I need
to put [in] some funds behind [them]. The best firms start by committing capital-investment
dollars in specific directions around customer value and customer impact, around
creating value, which is manufacturing or back-office systems, or enabling [systems].
By picking three buckets, I then immediately isolate some of the worst arguments
between the plant-committed guys and the customer-committed guys who start throwing
ROI around. By breaking them into buckets, you start to clean up the conversation.
For instance, if I have a $100 million to invest in capital projects, I say
60 percent is going to customers, 30 to value creation and 10 to enabling. Right
there I’ve gotten rid of a lot of most absurd argument. The second thing that
then occurs is those buckets have specific business goals they’re trying to
achieve-you know, improving employee productivity might be a good one in the
value creation [bucket], improving customer satisfaction in the customer facing
[bucket]. Then the third step is to prioritize projects based on their impact
on those objectives, some of which are ROI and some of which are soft.
Kalafut: Can I just add to that a little bit? A value creation index
which links measures like those you just mentioned back to things like market
value, and getting estimates of how much a change of those will actually impact
bottom-line performance, is another next step in the actual handling and treating
those intangible drivers. Some of [those measures] are those that you mentioned,
which are where we find three or four proxies to represent each of those drivers,
and then link them back to market value so that they can be scored. [There’s]
quite a difference [between] being able to actually put together several of
those intangible drivers, particularly in this area of IT where so much of that
is previously considered immeasurable.
CIO Insight: How do you generate trust that these drivers, these indicators
actually work?
Kalafut: Well, this is a ten-year study of various organizations. We
compiled various measures that tend to work in certain industries. We correlate
them back to things like market value, [and] then watch them across time to
see how those measures change. So it’s not a one point in time analysis that
makes you decide that those measures are the correct ones, but actually watching
them for several years. Once we were confident that those indicators were working
and could be gathered for a variety of different industries and companies, we
were able to go back and then start designing unique models, adding in additional
characteristics for specific company types and build models around those, including
various intangible drivers that might be measured in disparate parts of an organization
but have never been pulled together into one model.
CIO Insight: Denise, we’ve talked about Wall Street and portfolios, about
separating out different kinds of value and analyzing them separately. Would
this pass muster with you sitting in the CFO seat?
Fletcher: Well, I guess I start off back at what Christopher Gardner
said earlier. I start off with shareholder value. That is the driver, nothing
else is. Then the next step is: What drives shareholder value? The only thing
that studies have shown actually truly correlate to shareholder value is free
cash flow. Free cash flow is an element in the ROI analysis. It is, as I said
earlier, subject to a lot of vagaries because you’re doing projections. In my
view, the whole key is not bucketing or anything like that. In my view, the
key is to start your process, [as] Joe Barkley said earlier, with a strategy.
Where are you taking the company? Once you’ve identified where you’re taking
the company, the next step is to identify the strategic drivers that are actually
going to shift the company to where you want it to go, and where you’re going
to increase shareholder value. Then you look at the strategic driver and you
say, what is the role of IT, what IT do I need? It’s not really the role, but
it’s really what infrastructure, what do I need to make the strategic driver
happy? Well, I need to have a CRM system, I need to have 250 salespeople, I
need to get rid of 25 number-crunchers, etc. Put all of that together. Once
you’ve put the together, you then say, what is the impact of this on my company,
and that has to be added to the budget. And if I can’t swing that because maybe
for the first couple of years I’m going to have negative free cash flow [before]
it’s going to turn around, I obviously have to get rid of some of the things
that I’ve been doing until now. Joe was alluding to [this] before, when he talked
about looking at how we do things every day in our business.
So as I see it, the benefit of going through a process like this, where you
identify the strategy first, is that IT flows in as an enabler of the strategy,
not as somebody who is running around trying to implement a specific thing.
Why do I want to have this enabler? ROI, and I have to really push back on this,
ROI is just a tool. As you say in IT, if the input is lousy, the output is lousy.
Barkley: Garbage in, garbage out.
Fletcher: Thank you, sir. But garbage in, garbage out is also true of
ROI. Again, going back to what Joe said, whether it’s ROI [or] any number of
other techniques, you just really need them because you have to set priorities
at some point. But if they’re all going to be positive and you’re a very cash-flow
rich company, you can afford to do all of them, and sometimes your gut tells
you the numbers are there.
CIO Insight: Is there an example from companies where this works very well?
Fletcher: Well, in my last position at MasterCard we did implement a
process that we called a two-phase budgeting process. Phase one was in the first
and second quarters of the year, where we really only focused on strategy. And
then the goal in the fall was that the numbers would flow from the strategy,
as opposed to the numbers driving the strategy, which is what I think a lot
of companies do. It becomes a mechanistic budgeting process. Who cares about
the decimal points? The CEO doesn’t make decisions based on decimal points.
The reality is, if you’re going to have an after-tax return and it’s a positive
return, unless you are bidding one project against the other, you really don’t
care if it’s 15 percent after-tax or 25 percent after-tax. And I’ll be honest
with you: When I get a number like 3,000 [percent ROI] or whatever Christopher
was talking about, I do the exact same thing Christopher does-I slash. I say
that’s nonsense. I discard it and I say, okay, it’s greater than my cost of
capital, and that’s really all I care about.
CIO Insight: I want to continue going around the table getting reactions
and specific suggestions on what we need to do to do a better job of looking
at value. What’s the optimal approach that we need to take? Tom, let me get
your response.
Murphy: Well, I’ve kind of articulated what our approach is. We have
aligned our strategy, but each element of that Jumpstart initiative does go
through a review process. We do go through a process of evaluating value. We
do some bucketing. We have a lot of safety, environmental and regulatory issues
as a company that we have to address. Obviously, those become high-priority
items that go onto the project list with relatively little review. We do have
revenue-benefit, cost-efficiency and business-expansion projects that get a
little bit more financial focus in terms of ROI than some other projects. Then
we have our strategic and goodwill projects, and under goodwill we have guest
and employee satisfaction, which are harder to measure with pure financial elements,
but we spend a little bit more time focused on business objectives. And those
go into the formal evaluation process, and we do go back six months later and
determine if they, in fact, had a return. Were the objectives met? Were the
actual financial goals met? We meet on a quarterly basis with all of the senior
officers of the company and review the project prioritization and modify it
as we go. And then, depending on how well the company is doing, we can pull
back on our capital investment or increase the capital investment relatively
easily, based on our project portfolio process and prioritization process that
we use.
There’s a very strong sense of engagement on the part of the business leaders
that they are part of the decision-making process, that they’re part of the
valuation process as opposed to IT being in that cat bird seat.
Hirji: I think the cardinal rule has to be to keep it simple. Anyone
can take something and make it overly complex. It looks very precise, tons of
man-hours get chewed up doing it, and everyone feels better, but the reality
is you’re no further ahead. So the real genius is turning decision-making and
making it simple. So let me walk you through how we do it. We think we do a
reasonable job.
It’s a little strange to me to have a conversation about IT not being connected
with the business, because in our case IT is the business. So there’s never
a disconnect. We are the product, therefore, for us to not be connected with
the business is impossible. Second, my role, for example, is half shaping and
executing the strategy of the firm, and half running the IT organization. So
there is no major decision that is made without IT being at the table; similarly,
IT would not make any major decision without the business being there. We never
have the situation where the "R" and the "I" are in different
places. Third, we have very specifically identified what our strategy imperatives
are. So getting back to making sure IT is aligned with the strategy, in our
case, is very simple. We are the low-cost producer with the highest margins,
therefore, priority Number one is things that bring that cost structure down.
Number two, we provide the best tools in the industry for our traders, so we’re
always looking at creating the next generation toolset. And, third, we’re creating
new tool sets. So, again, we have three very clear goals. We can align all the
projects against one of those three goals. If it doesn’t meet one of the three
goals, it’s off the table.
The next thing that happens is [that] every single week the business and technology
folks get together and run through the [project] prioritization. They basically
list it one through ten in terms of what they want. We do that weekly. Why?
Because our projects are small [and] because we don’t bother trying to estimate
something that’s a year or two out. The marketplace changes; what was important
last week may not be important this week. The one rule we have is [that once]
a project is being coded, it’s too late to call it back. But that creates the
trust in the business; if you change your mind about what you want to do, you
are at worst four weeks away from getting it implemented. Before any project
gets on that list, it has had a validation or business case; it has a number
of things in it that say this is why it’s important. Sometimes it’s ROI, sometimes
it’s customer [service], sometimes it’s a bunch of things. The important point
is that that process gets both the IT folks and the business folks to say, we
jointly believe this should be on the list and it’s important, and we think
it should be number X out of 50 or whatever the number is. That’s why you do
the process. It’s irrelevant what the number is. What’s more relevant is getting
alignment.
CIO Insight: Does this begin to address some of the concerns you have about
bringing back the human element to looking at value?
Riazi: I’m not certain. I mean, I’ve heard a lot of great ideas here.
We always fall back on our comfort zone, and I think in a debate we fall back
to our analytical approach and a logical approach to a problem which I don’t
find to be logical. I find it to be emotional. I think, going back to your question
in terms of ROI, the best way to make it work is to look at an organization
organically. And it goes back to what Denise said: You have to look at it strategically,
[you have to] look at it organically, and acknowledge [ROI] will help, but it
shouldn’t drive it. The reason it’s failed miserably is because it’s been the
driver in many ways, and it’s been sold as a driver. And that’s where it becomes
very difficult for it to deliver.
I [will] just close with one story with my nephew. I went to buy him a Halloween
costume. He wanted to be Superman, so I bought him a Superman costume-he’s eight
years old. I took the costume home, and I looked in the back, and it said, "This
does not enable you to fly." (laughter) And that’s an important lesson
for us. The message out there is this technology, all this wonderful stuff is
going to make you fly. It doesn’t, and it won’t. And somehow we need to come
to terms with that. I’m hoping you guys find the answer because all the CIOs
are waiting for that answer, and the CFOs who are dishing out a lot of money.
Hirji: On that point, if anyone argues that technology confers competitive
advantage, I think they’re deluded. It’s not the technology, it’s the people.
Riazi: That’s all of us. I mean, the majority, has been sold [on IT]
this way that it’s going to make you fly.
Hirji: What I would argue is: If we have the best technologists, they can’t
help but deploy the technology better, cheaper, faster, smarter than anybody
else. And so the ultimate focus here is get the best people in place, they’ll
take care of it after that. So I agree, but I think the challenges are about
building those kinds of cultures, building those kinds of organizations.
Riazi: The challenge, I think, is a business challenge, it’s not ours as
CIOs. The business has to say I own this and this is my problem. Don’t go putting
in a big ERP and see me a year later. Alignment is not only IT to business,
it’s [also] business-to-business alignment, because
