ROI Roundtable

By CIOinsight  |  Posted 11-07-2003
  • Joseph Barkley, Vice President and CFO for Global Information Technology, American International Group Inc. (AIG)
  • Bobby Cameron, Principal, Forrester Research Inc.
  • Denise K. Fletcher , Managing Director, FA Group
  • Christopher Gardner, Partner and Cofounder, iValue
  • Asiff Hirji, Executive Vice President, CIO, Ameritrade Holding Corp.
  • Pamela Cohen Kalafut, Director, Intangibles Practice, Cap Gemini Ernst & Young
  • Thomas H. Murphy, CIO, Royal Caribbean Cruises Ltd.
  • Atefeh(Atti) Riazi, Senior Partner and CIO, Ogilvy & Mather Worldwide
  • Julie A. Schwartz, Vice President of Research, Information Technology Services Marketing Association

CIO Insight: Let me start with the first question: What is ROI? What does it mean? Does everyone agree on what it means? Atti, could you start us off on that? How do people usually define ROI?
Atefeh(Atti) Riazi: It depends who defines it. Mostly, the CFOs define what return on investment is, and it's defined from the financial terms. I think a lot of what we do in IT has changed and has evolved. For example, you cannot really use ROI in the way we know it in knowledge management. There are so many areas where you cannot apply the financial point of view to it because the return is not necessarily financial.
So who's defining it, why is it being defined? I personally, and I've shared this with you, I would say most CIOs are so fed up with the vendors who come in and sell this virtual vaporware with this great return on investment, which is nonexistent. You're kind of pushed to get to the CFO's office and say here, yes, there is an ROI. And I refuse to do it.

CIO Insight: We'll get back to talking about the vendors later. Joe, you've been a finance executive, working with IT organizations for many years. Is there a clear definition of ROI?
Joseph Barkley: The answer is yes and no. ROI is an accounting indicator that CFOs are comfortable with, because they know how to define return on investment, there are several mathematical formulas to get to that, and it's a number.
If you look at the value of IT and tie it to a financial number, that analysis is generally doomed to frustration and failure. ROI is merely one indicator that you can use to rank the project. One of the problems we have with ROI is we can't find it after we finish the project, because the project and the effect are melded into the rest of the budget and the operation. We haven't figured out how to go back and find it, verify all of the savings supposedly or all of the effect. Unless you've got a discrete business, by the way; then you can do it.

CIO Insight: But is there a specific definition of ROI within the finance?
Denise K. Fletcher: Yes, of course there is one. However, just like every other kind of modeling, the input is critical to the output. So if you start off your analysis [by] defining a project very narrowly, it is not likely to be representative of the investment.
Let me explain what I'm saying. If you look at a project, for example, [and say] you need to add storage capacity, and you say it's going to cost X number of dollars [and], I'm going to get Y return on this project, to me the weakness is that you have to step back to be able to get to the right place. You have to step back and say: How does this fit into the strategy of the company? You can't separate an IT project any more than you can separate a marketing project from the overall strategy. If you step back and you look at the overall strategy, I think that you will find that just about every project has a purpose and a return.
To be very specific, I was CFO of MasterCard until a little while ago. One of the major investments that we decided to make was to invest in brand. Investing further in brand meant that we were obviously going to get a lot more response from customers. But to be able to deal with that, we needed to add a lot of storage capacity. Now, I can look at my investment in brand and include in that investment my storage capacity, which is going to give a very positive return, or I can look at it separately which is not likely to give me a very high return.
So I think the most critical thing is to go back to a word that [CIO Insight Editor-in-Chief] Ellen Pearlman used earlier: it's the alignment, the alignment of the investment with the overall strategy of the company. I think ROI is only a tool, and the tool is only as good as the input that goes into it and as good as the boundaries that are defined around it.

Bobby Cameron: May I make a comment?

CIO Insight: Go ahead, Bobby.
Cameron: I think the series of comments point out two really critical factors. The last one I think is the most important, and that is there is no such thing as an IT investment, with the sole exception of some pure infrastructure cost-saving kinds of things. The justification or the rationalization has to be a complete view, meaning the business and the IT parts. There is no such thing as a standalone IT business investment. The second is that the mechanism, whatever it is, that goes to [the] justification of an investment has to be one that sits with the overall business operation, meaning if ROI or MPV or EVA or whatever is used for business investments, that should be what's used for IT. The flip side, which is frequently the case, is that IT is held to criteria that the rest of the business isn't held to. So, if IT is asked to do ROI and the rest of the firm isn't, then CIO's feel like they've run against the wall.

CIO Insight: I'd like to get back to that, Bobby, but let me get Tom Murphy on the phone. Tom, you told me that ROI has to be very closely connected to alignment as well. Could you comment?
Thomas H. Murphy: Well, I agree with Denise and Bobby, and our approach at Royal Caribbean is [that] we have a very clear business strategy that I've actually coauthored, and we have then aligned the IT strategy to support all of those business elements. We've laid out a $200 million investment called Jumpstart that has been approved at the board level with no ROI.

CIO Insight: What is Jumpstart?
Murphy: Jumpstart is our core business systems rewrite over the next three years. There are elements of employee, guest and supply chain to all of those, and we have component projects that make up that investment, and that is aligned exactly with the strategy that the business has put forward. In fact, the board has accepted that the business cannot achieve their strategies if we do not execute these IT projects that are really business projects.
But each of those projects, as they come up, are prioritized by the business and justified by the business for their value. In some cases those [justifications] are very ROI-driven, but in most cases they're strategic or fit into this jumpstart model. They still have to go through an approval process, but ROI is not the sole measure. There are many measures of that and, again, it aligns very neatly with the strategy that's been laid out for the next three to five years by the business.

CIO Insight: I'm hearing that everyone's emphasizing the importance of alignment. Let's look at a numbers themselves. You were telling me about the ROI numbers that themselves are often subjective and even illusory. Could you talk more about that?
Asiff Hirji: I think that there's a vested interest in pushing ROI from many areas, because it gives decisions which are by their nature difficult to substantiate, and difficult to get a group of people to align on, the look and feel of specificity without, frankly, very much specificity in many cases. If you look at the way ROI typically came across, anytime you have a set of inputs-this is back to Denise's point-that are subjective in their nature, the measurement is, even though it's mathematical, still a subjective measurement. So if you have any sort of terminal value, if you have any sort of any long-term profit stream that's baked into there, you will create something that has this sort of taste and smell of precision without any of the actual benefit of precision. Anyone [who] comes to the table and says, yeah, this is a big number, it's bigger than that number, definitely let's go for this, [is] deluding themselves.
[The] objective here is how to make better resource allocation decisions. Whatever works for your firm, you have to find a way of matching the technology decision-making to that process. If your firm throws darts at a board, great, keep doing that for technology because that's part of how you make the technology decision-making in line with the decision-making of the business. And I think, again to the point that Tom was making, that better companies will have a very tight alignment. There's no such thing as a technology prioritization. It's business prioritization across various business lines, that flows to a set of projects, which flows to a set of initiatives. The way we do it at Ameritrade, every six months we revisit the macro level allocation across all the businesses. That's not just technology, but marketing, HR, the whole bit.

CIO Insight: Chris, do you accept that it's impossible to have objective ROI measures?
Christopher Gardner: No, having written a book on the topic, we have strong opinions about this, and everything that's been said today is absolutely what we've encountered. It's a very hard problem to solve. There are problems of isolation, alignment, the economics of companies vary, so the ROI definition varies from company to company. In fact, ROI in my mind is more of a proxy for a broader issue which is what is the value of IT, and how you measure that can be done in quite a variety of ways, some of which follow financial guidelines, [and] some of which are internal indices that are more meant to be something that correlates with value.
There's a bunch of additional problems with ROI. One problem is that the inputs tend to get done poor, that there isn't enough time assessing demand in a rigorous way. There isn't enough time looking at alternative designs and assessing those. The economics of the business are not assessed and analyzed so that the IT system's effects on the economics of that business are clear.
I think also, aside from kind of the hard work in getting the inputs right, the numbers can be dangerous. If they're not done properly, usually they're wrong, and then people get held accountable to numbers that are wrong. And so you have this situation where, frankly, I think there's an obligation to get the numbers right if you're going to hold people accountable, and the fact that the numbers can be dangerous is another issue here. But the inputs are the most critical thing to get by here.

Julie A. Schwartz: I'm listening to the conversation, and it strikes me that there still might be too much emphasis on the numbers, the ROI tool and calculation itself, and not enough on the process. Chris, I think you're absolutely right about the fact that sometimes the numbers can be wrong, and as part of the process what's missing is this test for reasonableness. I'm on the panel representing the vendor point of view, and one of the things that the vendors can be very helpful with, by working with their clients and their customers, is getting benchmark data together so they can build a database to help companies who are struggling with this problem to look at what is possible, what is reasonable. The other part of the process that's always missing is the postimplementation evaluation, going back and saying, okay, how were our numbers? Did we reach what we were going to achieve? What we found is that [in] very few, you know, maybe 20, 30 [percent] of the projects, [do] people go back and say, well, what did we do with ROI based on what our inputs were and what we got out? The problem is, we're not learning. There's no learning from the experience, so we're not improving the process in any way, shape or form.

Gardner: It's actually borne out in your CIO Insight survey. ROI analysis is done at the very beginning of a project, but after that, there's very little effort to follow through.

Cameron: Let me give you just another data point, some data we at Forester gathered at the start of this year. The numbers bear out very clearly what Julie just quoted. We saw on average only 37 percent of projects were measured postproduction, but the interesting thing is that when we took that sample [of] senior IT decision-makers at 174 $1 billion plus North American firms, and broke it into quartiles-in other words, those who did the most down to those who did the least measuring postproduction-what we found is that the top quartile measure 86 percent of their projects postproduction, 45 percent the second quarter, third quarter 13 and the bottom 1 percent. So my point is that those who learn how to measure and learn how to do it well do probably as much as they possibly can.

CIO Insight: I want to go back to Atti. You were talking before about vendors. We heard from Julie how they can help and what they can do. But do you feel that, in reality, they're helping as much as they could or that they're actually part of the problem?
Riazi: I think some of the way they've sold IT has been the ROI concept, and I think they are part of the problem in many ways because they never talk about the real issues. They come in, they sell you a concept that just looks terrific and wonderful, and they promise the ROI without sharing the issues related to change in the organization with the CFOs. The CIO becomes the therapist or the catalyst to make that change happen.
We are here discussing a very irrational, illogical issue in a very logical way. And that's the disconnect I feel, that's the lack of alignment. We are talking about people, and when you implement technology, you change the way they work. And that is very irrational. The reason ROI fails is because we forget these are people who are going to worry about their jobs and their training and experience, and what's going to happen to me. I haven't seen any research data or statistics on what happens there. Because the analysis and the statistics, all of them, are really a logical view: The process was very long, we collapsed it; the hierarchy was high and we collapsed it. Meanwhile, hundreds of people lost jobs, but that's irrelevant; that's not a logical argument.
So I think between the vendors and us CIOs there is a responsibility to look at the failure, and the failure is not a logical discussion. It is a discussion of what happens in an organization when that technology changes the way that organization works, and it's a lot of soft issues that happen. The biggest thing is what's going to happen to me, and that is going to stop your ROI. It's going to stop the return because [people] are not going to want to change. But the vendors have underestimated the importance of that. They too have approached [technology changes] in a logical way. You put this ERP and it's going to be great and it's going to be less head count and more efficient and better-you put the CRM in and you're going to automatically increase customers. And it's not real.

CIO Insight: Julie, I'd like you to respond, and then, Pam, this would be a wonderful moment to get you in to the conversation.
Schwartz: I think you make some very good points, but what vendors have learned is that with very few exceptions they are not able to get the kind of partnership relationship with their customers that they would like, so that they could walk hand-in-hand with the customers through the process you describe of really understanding the change management that goes on in the organizations where their solutions are implemented. In many cases we don't know how. [Vendors] are engineers, and they're very analytical, and they don't know how to deal with that kind of ambiguity.
But I think it's important to keep in mind that there are really two aspects of how vendors use the ROI. One is that they're trying to make a compelling value proposition to get the CIO's attention. You know, you sit there, every week you get hundreds and hundreds of unsolicited e-mails and phone calls. And [vendors] need to find a way to get in so that they can tell you about their solutions. So one thing they are doing is trying to come up with the most compelling value proposition, and that might very well include ROI that looks like smoke and mirrors or vaporware to you.
The other reason they need to have ROI models is to help their clients and customers with an internal justification to build a business case. Some vendors are going to do this very well, and others are not going to do this very well at all. They're learning as well. But the most important thing is for the customers to really open [up] and say [to the vendors] we'll work with you so that we can collect the data, we can build the models, we can look at the assumptions, and do this in a consistent way so that you can do this with other customers, so that we can build a database so that we can track what's reasonable, so that we can start learning from our experience going forward so that we can make this a better tool. It's more of a process than an output.

Pamela Cohen Kalafut: I think that all makes sense. One of the things we get asked about the most is how this measure or others are going to help businesses get some sort of competitive advantage, which, as we all know, is almost impossible to sustain in the present economy, except through measures that might not always be recorded by businesses.

Fletcher: If I could just make a comment. It seems to me, if you start off with a premise that ROI is part of alignment and part of a corporate strategy, then it's very difficult to expect the vendor to even have an understanding of what is going to drive the return on investment. For one thing, the discount rate that is used on the return on investment must reflect the capital costs of that company, not of the vendor. That's the first weakness. Second, I don't know how a vendor could possibly understand all of those soft costs that Atti was talking about, the cultural costs. They're hard enough for internal corporate management to assess, let alone a vendor. I think the weakness, as I hear this conversation, of the approach is that when an outsider is looking at [ROI] they're looking at a little box, but when you look at an ROI calculation to do it right, you have to look, as I said earlier, to the whole [set of costs], which really drives the responsibility of the ROI to internal management and not to the vendor.

Schwartz: The vendor is there to help in any way that they can, but they cannot do [the ROI calculations] for the customer. They can provide data: Have you thought of XYZ? Have you thought of ABC? Other customers we have worked with have done it this way. But they can't do it. In fact, we have done research with customers and asked who do you want to do your ROI calculations, and the majority say that they're going to do it themselves, and a few are saying they're going to do it with vendors helping them. But for the most part, they're going to do it themselves.

CIO Insight: I have a question. Is ROI, in general, a symptom of a bigger kind of problem: a lack of trust between the business side and the IT side, and [of] being able to prove that a technology investment is going to return something to the company? If there was better communication, perhaps there wouldn't be so much focus on that. Is it a reflection of that, or [of] something else going on in the enterprise?
Barkley: I'd like to get into that first. To quote one of our former presidents of our company, he's always fearful of "the system that ate Chicago," because systems tend to grow. So you're right, there's a fair mistrust. There's a fair mistrust of the financial analysis capabilities of information technology people. Don't take that the wrong way. And there's a fair mistrust of the technical competence of the financial people, which is one of the reasons there's now people like me in our information technology [organization]. Because at the end of the day, the CIO and the CFO have to agree on the strategy, agree on the costs, and agree on what we're going to deliver. But there's a fair amount of mistrust.

Murphy: I think the trust issue goes beyond IT and finance. Our finance group requires ROIs for any capital expenditure beyond $150,000. That includes building a new ship, which is a little bit like putting in a custom application. It's hard to nail it down. It's a physical asset, but it has a tendency to morph and grow and become more expensive as people have new, clever ideas, and a lot of it comes down to effective process and project management. I think the trust part comes in that once I, as CIO, have established that I have a methodology, and my team has process and discipline, for doing these things effectively, that changes the conversation. But I don't think that's any different than any other business unit that has significant capital requirements to do their business.

CIO Insight: Asiff, you had a comment?
Hirji: I think the sentiment in that question [about lack of trust] is right. I think IT professionals probably brought this upon themselves in some way. There was a study about how many major systems initiatives actually deliver, and the vast majority of systems fail. When I was a consultant, it was not uncommon to have projects of $200 million to half a billion dollars in size, the vast majority of which failed. Forget didn't deliver ROI, didn't get turned on. [Projects] got three-quarters of the way through implementation, and didn't get turned on. So it was easy to calculate ROI on that-zero. I think those days, thankfully, are dead, and that we collectively have learned a lot. But if you have a process by which your business allocates resources, checks progress on the projects, makes effective decisions about whether they're proceeding or not proceeding properly, reallocates and delivers shareholder value or earnings per share or whatever it is you're aiming for, it doesn't matter if you use ROI or not, it doesn't matter what tool you use to do that, so long as the process is working and you as the management team are working effectively with it.
I think ROI is great for vendors, and I don't blame them for using that. It's an effective mechanism. But if you're going to use it, you better use it in context of the way your organization makes decisions and in alignment with the way your organization makes decisions. It's like code: It's only as good as the inputs. You can write the best code in the world, if you feed a bad input, you'll get bad output.
So, there's too much emphasis on ROI as a measurement mechanism, and there's not enough on what's it about. Go back to that study that I think our friend from Forrester, Bobby Cameron, was quoting, he said top-tier companies do track ROI and they do track return on investment and so on. But I think CIO Insight's study said that people have different definitions of ROI. So my takeaway from that is companies that set some goals in advance, measure the project's process, then measure against those goals what the project actually did, they learn and they get better. It doesn't matter whether you use ROI as part of that or something else as long as you are setting some goals, doing the project, checking it against the goals afterwards.
And in our experience at Ameritrade, the more you shorten the lifecycle of that, the better off you are. We will not do anything that is longer then 12 weeks. If we can't do it in 12 weeks, it's dead. It doesn't get off the ground. Why? Because as bad as we all are at estimating, if we're estimating from six to nine months out versus six to nine weeks out, we'll take the risk on six to nine weeks. It's just human nature.

Barkley: Let me just pick up. We worry about delivery. Once we approve the project, ROI [is] one of the indicators, but we're more interested in the alignment and technology for our strategy. We're interested in the delivery of the project in the budget, and don't be late. We worry about those two things more than anything else. The other thing we started to do this year is every time we develop or design a new application or a new project, we put a metric in there so we know if it's working. Let me give you an example. If you build an application that's going to do policy issuance and it's supposed to be able to issue a million policies a month and you run it and it issues a policy, [people] say it works. Well, it doesn't work. It only issued one policy. I want to see it issue a million policies. Then I know it works. We have not in the past had operational metrics. When we install them, we started to improve our delivery in our continued operations, and the value, by the way.

Fletcher: Well, a couple of thoughts. I think that there was a comment made earlier about specificity and what goes into the equation is kind of a little bit random. I'd like to answer that. In my view, ROI is a good tool as long as it is used consistently throughout the company for all kinds of different decisions, whether it's marketing, HR or anything else. You can't just say IT goes the ROI way but marketing does its own spiel. That doesn't work. Second, I think ROI is a very useful tool for allocating funds if you have a lot of demand for those funds, and you have to make a judgment call between one [demand] versus another. If you don't have that kind of situation and you have all the funds that you need in the world, I think free cash flow is a much better tool. It too, though, is subject to some of what we call lack of specificity or subjectivity because, let's face it, we're all human, and we're all going to do projections, and we all make the projections from the time in history of which we are living right now. If you ask people to make a projection now about revenue growth, they're going to give you one kind of projection, which would be very different from anybody's projections three years ago. We never have the ability to forecast cycles. It's always a straight line from whatever point we're in, or a hockey stick.
Where I'm going [with that] is I think the most important thing is always a strategic driver. Why are we making this investment? Is it because we want to increase knowledge? Okay, that's great, we want to have certain kinds of efficiency. Let's not kid ourselves that we need to have a return on investment. But if I have only $100 million dollars and I have to decide whether I'm going to go with Atti's project or I'm going to go with a marketing guy's project, and they each want $80 million, I can't give them $160 million if I've only got $100 million. So I've got to make some kind of an allocation decision. But if I don't have that challenge, as long as it's free cash flow positive, let's go do them both.

CIO Insight: Atti, I was fascinated by your comment on culture, essentially the soft issues. I wonder if you could elaborate a little bit about this. I'm hearing about credibility problems, I'm hearing different approaches to who has leadership here. We also hear a lot about rogue IT projects. I'm just wondering what this is a symptom of. Is this a leadership problem? How do we fix this?
Riazi: I wish I knew the answer. I've struggled with this issue the last 20 years being in IT and being a CIO. I think there's one reason they say the life of a CIO is three years, because the first year is infrastructure, the second year is begging for money to do ERP applications, and the third year is when you actually do it. And by the third year, you have the whole organization ready to shoot you. You learn two things as a CIO [to be] able to separate those who hate you from those who are undecided. (laughter)
This is a very complex issue, and I think, and I'm hoping through panels like this and the research companies [to] start going to the forbidden zone. We all hate to do [it]. We're all analytical IT, finance people, and I think we've lost focus on the real issues that I have faced in every project I have rolled out, every large project. I was training some of the finance people, I was in the room where they were being trained on a wonderful application, and three of them sat back and said but that's my job. I was showing them how it was going to be automated, and the light bulb went off, that's my job. Well, that was the job of another 50 percent in the room, but the light bulb hadn't gone off yet.
How do you get ROI? You get it through people. I mean, the biggest cost is people, so that's the first way you're going to go. Even if you don't let people go, you're going to make them more efficient, they're going to have to learn something new and there is the evaluation on that. I think we [have to] start looking at our failures, because our success has been abysmal. Our credibility is really shot. And that's why I said earlier I refuse to go to see my CFO with an ROI case because he's going to laugh at me and is going to kick me out of his office.

CIO Insight: So how do you survive? How does any CIO survive?
Riazi: I don't use ROI. I don't think CIOs should use ROI because the "R" isn't in my department; the "I" is in my department. The "R" is in the business which I have absolutely no control. So why should I use it?

Cameron: I disagree. I think you yourself pointed out an example earlier where the expense sits on both sides, where the user was sitting down and recognizing that's my job, the investment really is from them, too. They have to put time in; they have to change their process. Good technology has not nearly the impact [of] good process change, with or without technology to support it. So the bottom line, what I'm hearing you say is, not that ROI fails or that the projects fail or there's maybe a bad odor of IT history, it's that we in IT have sat alone and tried to do these things alone, and that really the cultural change is for us to become part of the business so that we relate to the rest of the firm like, let's say, a manufacturing plant's general manager does to the rest of the firm.

CIO Insight: Is ROI politics? Is it possible to say that ROI is a technique that business people use to put the onus and responsibility for IT failure on IT people? You come up with an ROI, you prove it to us, and if doesn't make it, it's your fault.
Murphy: I cannot imagine [such] a scenario where we would have a project go forward at Royal Caribbean, where I was developing the ROI. It isn't done. The business unit has to be accountable for the return on investment, except [for] a few real core infrastructure-types of initiatives, [and] even then I can flip back and show them how the value is going to go to the business. Now, you know, alignment is a big part, culture is a big part, trust is a big part. I have a personal relationship with every businessperson in this organization, and I am one of the businesspeople. I don't think people view me as that IT guy who doesn't deliver on projects. I'm in a partnership with all of these folks trying to drive the business to success. It's a fascinating conversation, and there's a lot of angst out there obviously, but I think a lot of it is in the relationships that you develop within the business and the trust and the process that you put into place.

Barkley: We require a business case on every project over $1 million that comes to my desk. Before it gets to my desk, it's signed off by the business CFO. Even after he signs it, I'll call him up and say: [Do] you understand what you signed? Do you understand how you're going to find a savings? Because I'm not the guy who's going to look for them, you are. We require that. In our company, the application development is decentralized; that is, we have CIOs in each of the businesses that are matrixed to my boss, the global CIO and also to the business executive, although we control all of the infrastructure centrally. So the businesses have a big stake in the strategy, what we're going to do, and the returns as well. Again, ROI is merely one of the indicators.

Fletcher: I was very concerned by a comment that Atti made which was that the return is at the unit, the "I" [in ROI] is with her. That would concern me because I think that that, to me, would be a little bit of a cultural disconnect between the IT organization and the rest of the organization. The most important thing to remember is that the "R" and the "I" are owned by the company.
So you need to get over the sort of silo thing, and you need to go, I think, to the models that both Joe and Tom are talking about, where you have a really strong alignment between the business units and IT, whether the IT is decentralized or centralized. But I think there are some pitfalls thinking that the "R" and the "I" are disconnected, because it all ends up in the shareholder value, and the shareholder value is the company as a whole. You can lose sight of that in the silos.

Gardner: I have to say I agree with everything that's been said here. I'm not doing it because I'm trying to form any consensus; I think everything here is true. I like Atti's comment about people problems. I mean some of these soft benefits or soft costs, how do you get at those? There are ways to do that, and we actually borrow techniques from the advertising industry to get at some of these people issues, primary market research where it's not a precise number but it's a good enough number, and you can start to see what the dynamics are between some of the choices and some of the design alternatives.

CIO Insight: But how do you get at the people issues? I just want to make sure I understand what those people issues are and how they relate to ROI.
Gardner: Let me just make a couple of other comments. One is that I don't think that we can escape this question of how we measure the value of IT, because management will always have a problem with IT so long as you can't measure its value. If you can't measure the value of IT, my sense is-[and] this gets back to your point about is there trust between upper management and IT-if you can't measure the value of IT, how can you claim management, and the very top levels of the organization, are held accountable for shareholder value?

Cameron: Is there a value for IT? I would argue there isn't. There's a value for business execution or delivery or value delivery, with IT as a component.

Gardner: Which is fine. You've got to look at IT in the context of the company, [at] what IT is doing to contribute to the overall value of the company. Maybe it's helping to create various competitive advantages. IT can be costly; that can create a capital barrier. There are skills involved, some of the people issues relate to IT skills, and that can give you can advantage. So, yeah, IT has to look at the broader context that it sits in, and you've got to understand how the company makes money so that the ROI [analysis] that you're performing ties to the way the company is actually making money. But I think there are ways to get at some of these things. I think there's also an issue around precision. [With] ROI, I think, sometimes there's a number that people are looking for [with] several decimal points. I remember seeing a vendor ROI analysis that came up with a number like 3,287 percent for some system, and that's just not the way to think about these things. Initially, you go in, you might do a back-of-the-envelope type of calculation, and just see if it even makes sense to pursue further. And then over time, as things proceed and get more serious and maybe more money is involved, you make the effort to do the primary market research, to do the careful estimate and adjust for the cost of capital and risk. You know, it's amazing to me that [in the] companies that we've served, there's a lot of talk and focus on ROI, the return part of the equation, but you need to look at risk and return together. Risk is completely left out of the equation. For those things that you can't quantify and are very uncertain and ambiguous, one can at least make an estimate of what's the risk. Is it a code red? Is it something that could affect the entire success of the project, or will it have a negligible effect? You can at least do that kind of thing.

Kalafut: I just wanted to get at the question in terms of the human side [of ROI], because the [human side of technology] is one of the things we talked about in [our book] Invisible Advantage and as part of our value creation index. We know there is no magical power in the technology itself. But in terms of the human side of technology, we have the work processes that go into it which are run by the people that actually have the power to create the strategies around it, and [who] motivate others to actually use that [technology] and decide that they're not going to be replaced by those processes. Also, in terms of work place organization, creating the jobs and responsibilities, the change in how the training comes into play to actually absorb people with new talents within the work place, has been one of our measures. Erik Brynjolfsson and his colleagues at MIT have studied companies' investments in IT, and found that they're almost invariably accompanied by much larger investments in intangibles, such as training in the workplace, and also by empowerment of employees as well. We go through a variety of examples, but all of it actually comes down to how you empower people to use those technologies rather than the technologies themselves.

CIO Insight: I just want to step back now. We've talked now for close to an hour about what the problems and the issues with ROI. The pressure is there to come up with an ROI, but it's still difficult to calculate. It seems to me that this is still something that we need to do, at least as part of the larger issue of value, particularly in the financial community. Does that sound right?
Barkley: It's slightly different. Maybe you're being overly focused on ROI. Here's why. We suddenly realized in looking at our budget process over the last two years [that] we spend 85 percent of our time on 30 percent of the budget. Projects are important, capital investment is important. But 68 percent of our money to is to keep the lights on and the business running. We hadn't done a good job at budgeting and analyzing that. So we had to figure out a way to do a better job at budgeting lights on, and aligning lights on with the strategic intentions of the company. So we've actually changed our budget process where we do the lights-on budget first, and then late in the budget process, after the businesses have done their budget, we get the project budget from the businesses, and we go through a strategic-alignment value analysis where we measure, among other things, internal rate of return, business value risk and dependency. We do that in the last few weeks of the budget [process]. While [the project budget] is important and we need to have it to justify the projects, we've got a large part of the business that we realize we hadn't been paying attention to, and we have to analyze that too.

Riazi: My problem is that I think we have sold technology to the business in isolation. Here it is, you put it in, and it's going to be wonderful. I call it the diet pill: you take it, and you lose 50 pounds, and you don't have to worry about anything else. That's the underlying problem.

Barkley: Do you know where that pill is? (laughter)

I think technology has a great deal of ROI, it's just our approach to it, our mind-set [that is a problem]. The fact is [technology] enables you to do so many different things. It enables you to streamline your operation, if that's how you approach it. [But] if you approach it that we're going to dump this and it's going to be great and it's going to give us the ROI, it's going to fail. So I'm not questioning the value of ROI. I believe it is [valuable] and I believe we haven't even scratched the surface. But there are other issues.

CIO Insight: What I'm hearing is we need to take ROI value out of isolation for the rest of the business. We need to make this part of a process. We need to use that process to build trust. And that the real purpose of ROI is to make allocation decisions; that ultimately is really what it's about. So, how do we transform our narrow, isolated approaches to ROI to something that's going to really get at the question of value, and is really going to help us make better decisions? Where do we start on that? Chris, can we learn something from Wall Street and how Wall Street looks at valuation of companies?
Gardner: Absolutely. My feeling is the top of the company is measured based on Wall Street style valuation models, and that should be reflected elsewhere in the company. Those [valuation models for IT] can be built, they're complicated and they're hard, but it can be done, and at least you have some assurance that your analysis ties back to something that is meaningful from a top executive standpoint.

CIO Insight: What would be the basic things needed to achieve that?
Gardner: Well, you need the free cash-flow approach that Denise was mentioning. It's ultimately about cash flow. We've talked about ROI here, but my feeling is ROI is more of a general term that encompasses the concept of value for IT. The free cash-flow approach, there's strong evidence that shows it correlates with share price better than earnings per share and a bunch of other things. But this gets back to the inputs. If you've got a system that involves a lot of customer adoption, you need to understand the customer's response to the design that's been proposed. You've got to work out those costs in detail. You have to be complete about your costing. You've got to make sure training is in there and change management, [and the] inefficiencies, the delays and that kind of thing. And some of these may not be known going into it, but you can see, well, what if we're off here by 50 percent or 10 percent or 100 percent? Does it make any difference?
You can start to create something that you can use to manage the whole process of building a system. What we're talking about isn't something you just use at the beginning of a project to decide whether to invest in it or not; it's something of a tool that you use throughout the lifecycle of a project. So during the development you would know what are the get-rights. You know, if you've got a narrow market-opportunity window, schedules will be paramount. You can show that and model that. Then [there's] the operation of the system itself. This is really something that should be an operational tool and tie back to operational measures.
It's hard, and there's a lot of uncertainty here. The key is to always ask questions and challenge it. Credibility is low right now, but a tool like this should start to predict that some projects are not worth doing and some projects are worth doing. IT should start to be able to show that the business might be recommending projects that don't make sense. This is a politically difficult kind of thing, but you should start to see some projects being turned down. A test is: What am I seeing coming out of this? Does it calibrate with reality? Am I getting some responses that look halfway credible? Can someone show me the evidence, make the case, you know, sometimes you have to put it on the line.

CIO Insight: Bobby, what do you think? Is looking at IT investments kind of the way investors look at portfolios a helpful way to look and think about ROI? Is that something we all need to be doing as part of changing and transforming ROI?
Cameron: Let me make a couple of comments before I answer that directly. One is that it really is about the process. No matter what you do, if the process doesn't involve everybody fully, meaning business and IT equally, and the justifications don't cover everything, nothing works. So just put that on the table and push it aside. The second overall statement is that we've focused on ROI as if it were the be all and end all, but not everything either merits [justification] or can be justified, and so we need to make sure that we put some clarity there. Most firms that identify minimum pain around prioritization isolate strategic mandatory and operational investments into those categories where strategic justification is different from mandatory, tax regulatory, biggest-customer-tells-me-I-got-to [justifications], and neither of those is justified based on ROI.
ROI is really calculated for a portion of the investment stream that has to do with things that are justified based on changes to operations. In other words, I expect a business impact. And then there are firms who are committed to ROI who also justify projects based on softer metrics, EMC being one, for instance, where they use soft metrics for things that might improve customer satisfaction but would be very difficult to prove an ROI.
So now that's sort of laid the ground for what I'm about to say, which is that most firms want to invest, as someone said earlier, to drive competitive advantage and improve stakeholder value. Frequently firms do their darndest to understand what the strategies are, what the directions are that they need to accomplish these goals. The problem is [that] there's no linkage between that planning and thinking and investments usually in the prioritization process. What I've seen emerging is, in fact, a portfolio mechanism, meaning we look at everything as a group of investments and we compare them to each other, and the process attempts to adjust the prioritization [of] strategic needs.

CIO Insight: So what are the first steps then that we need to do to change and improve the way we do ROI?
Cameron: Well, ROI is best used when it's a relative comparison tool. It has no absolute characterization. What I mean by that is if I want to adjust the direction of my spend to parallel what I'm trying to accomplish, I need to put [in] some funds behind [them]. The best firms start by committing capital-investment dollars in specific directions around customer value and customer impact, around creating value, which is manufacturing or back-office systems, or enabling [systems]. By picking three buckets, I then immediately isolate some of the worst arguments between the plant-committed guys and the customer-committed guys who start throwing ROI around. By breaking them into buckets, you start to clean up the conversation. For instance, if I have a $100 million to invest in capital projects, I say 60 percent is going to customers, 30 to value creation and 10 to enabling. Right there I've gotten rid of a lot of most absurd argument. The second thing that then occurs is those buckets have specific business goals they're trying to achieve-you know, improving employee productivity might be a good one in the value creation [bucket], improving customer satisfaction in the customer facing [bucket]. Then the third step is to prioritize projects based on their impact on those objectives, some of which are ROI and some of which are soft.

Kalafut: Can I just add to that a little bit? A value creation index which links measures like those you just mentioned back to things like market value, and getting estimates of how much a change of those will actually impact bottom-line performance, is another next step in the actual handling and treating those intangible drivers. Some of [those measures] are those that you mentioned, which are where we find three or four proxies to represent each of those drivers, and then link them back to market value so that they can be scored. [There's] quite a difference [between] being able to actually put together several of those intangible drivers, particularly in this area of IT where so much of that is previously considered immeasurable.

CIO Insight: How do you generate trust that these drivers, these indicators actually work?
Kalafut: Well, this is a ten-year study of various organizations. We compiled various measures that tend to work in certain industries. We correlate them back to things like market value, [and] then watch them across time to see how those measures change. So it's not a one point in time analysis that makes you decide that those measures are the correct ones, but actually watching them for several years. Once we were confident that those indicators were working and could be gathered for a variety of different industries and companies, we were able to go back and then start designing unique models, adding in additional characteristics for specific company types and build models around those, including various intangible drivers that might be measured in disparate parts of an organization but have never been pulled together into one model.

CIO Insight: Denise, we've talked about Wall Street and portfolios, about separating out different kinds of value and analyzing them separately. Would this pass muster with you sitting in the CFO seat?
Fletcher: Well, I guess I start off back at what Christopher Gardner said earlier. I start off with shareholder value. That is the driver, nothing else is. Then the next step is: What drives shareholder value? The only thing that studies have shown actually truly correlate to shareholder value is free cash flow. Free cash flow is an element in the ROI analysis. It is, as I said earlier, subject to a lot of vagaries because you're doing projections. In my view, the whole key is not bucketing or anything like that. In my view, the key is to start your process, [as] Joe Barkley said earlier, with a strategy. Where are you taking the company? Once you've identified where you're taking the company, the next step is to identify the strategic drivers that are actually going to shift the company to where you want it to go, and where you're going to increase shareholder value. Then you look at the strategic driver and you say, what is the role of IT, what IT do I need? It's not really the role, but it's really what infrastructure, what do I need to make the strategic driver happy? Well, I need to have a CRM system, I need to have 250 salespeople, I need to get rid of 25 number-crunchers, etc. Put all of that together. Once you've put the together, you then say, what is the impact of this on my company, and that has to be added to the budget. And if I can't swing that because maybe for the first couple of years I'm going to have negative free cash flow [before] it's going to turn around, I obviously have to get rid of some of the things that I've been doing until now. Joe was alluding to [this] before, when he talked about looking at how we do things every day in our business.
So as I see it, the benefit of going through a process like this, where you identify the strategy first, is that IT flows in as an enabler of the strategy, not as somebody who is running around trying to implement a specific thing. Why do I want to have this enabler? ROI, and I have to really push back on this, ROI is just a tool. As you say in IT, if the input is lousy, the output is lousy.

Barkley: Garbage in, garbage out.

Fletcher: Thank you, sir. But garbage in, garbage out is also true of ROI. Again, going back to what Joe said, whether it's ROI [or] any number of other techniques, you just really need them because you have to set priorities at some point. But if they're all going to be positive and you're a very cash-flow rich company, you can afford to do all of them, and sometimes your gut tells you the numbers are there.

CIO Insight: Is there an example from companies where this works very well?
Fletcher: Well, in my last position at MasterCard we did implement a process that we called a two-phase budgeting process. Phase one was in the first and second quarters of the year, where we really only focused on strategy. And then the goal in the fall was that the numbers would flow from the strategy, as opposed to the numbers driving the strategy, which is what I think a lot of companies do. It becomes a mechanistic budgeting process. Who cares about the decimal points? The CEO doesn't make decisions based on decimal points. The reality is, if you're going to have an after-tax return and it's a positive return, unless you are bidding one project against the other, you really don't care if it's 15 percent after-tax or 25 percent after-tax. And I'll be honest with you: When I get a number like 3,000 [percent ROI] or whatever Christopher was talking about, I do the exact same thing Christopher does-I slash. I say that's nonsense. I discard it and I say, okay, it's greater than my cost of capital, and that's really all I care about.

CIO Insight: I want to continue going around the table getting reactions and specific suggestions on what we need to do to do a better job of looking at value. What's the optimal approach that we need to take? Tom, let me get your response.
Murphy: Well, I've kind of articulated what our approach is. We have aligned our strategy, but each element of that Jumpstart initiative does go through a review process. We do go through a process of evaluating value. We do some bucketing. We have a lot of safety, environmental and regulatory issues as a company that we have to address. Obviously, those become high-priority items that go onto the project list with relatively little review. We do have revenue-benefit, cost-efficiency and business-expansion projects that get a little bit more financial focus in terms of ROI than some other projects. Then we have our strategic and goodwill projects, and under goodwill we have guest and employee satisfaction, which are harder to measure with pure financial elements, but we spend a little bit more time focused on business objectives. And those go into the formal evaluation process, and we do go back six months later and determine if they, in fact, had a return. Were the objectives met? Were the actual financial goals met? We meet on a quarterly basis with all of the senior officers of the company and review the project prioritization and modify it as we go. And then, depending on how well the company is doing, we can pull back on our capital investment or increase the capital investment relatively easily, based on our project portfolio process and prioritization process that we use.
There's a very strong sense of engagement on the part of the business leaders that they are part of the decision-making process, that they're part of the valuation process as opposed to IT being in that cat bird seat.

Hirji: I think the cardinal rule has to be to keep it simple. Anyone can take something and make it overly complex. It looks very precise, tons of man-hours get chewed up doing it, and everyone feels better, but the reality is you're no further ahead. So the real genius is turning decision-making and making it simple. So let me walk you through how we do it. We think we do a reasonable job.
It's a little strange to me to have a conversation about IT not being connected with the business, because in our case IT is the business. So there's never a disconnect. We are the product, therefore, for us to not be connected with the business is impossible. Second, my role, for example, is half shaping and executing the strategy of the firm, and half running the IT organization. So there is no major decision that is made without IT being at the table; similarly, IT would not make any major decision without the business being there. We never have the situation where the "R" and the "I" are in different places. Third, we have very specifically identified what our strategy imperatives are. So getting back to making sure IT is aligned with the strategy, in our case, is very simple. We are the low-cost producer with the highest margins, therefore, priority Number one is things that bring that cost structure down. Number two, we provide the best tools in the industry for our traders, so we're always looking at creating the next generation toolset. And, third, we're creating new tool sets. So, again, we have three very clear goals. We can align all the projects against one of those three goals. If it doesn't meet one of the three goals, it's off the table.
The next thing that happens is [that] every single week the business and technology folks get together and run through the [project] prioritization. They basically list it one through ten in terms of what they want. We do that weekly. Why? Because our projects are small [and] because we don't bother trying to estimate something that's a year or two out. The marketplace changes; what was important last week may not be important this week. The one rule we have is [that once] a project is being coded, it's too late to call it back. But that creates the trust in the business; if you change your mind about what you want to do, you are at worst four weeks away from getting it implemented. Before any project gets on that list, it has had a validation or business case; it has a number of things in it that say this is why it's important. Sometimes it's ROI, sometimes it's customer [service], sometimes it's a bunch of things. The important point is that that process gets both the IT folks and the business folks to say, we jointly believe this should be on the list and it's important, and we think it should be number X out of 50 or whatever the number is. That's why you do the process. It's irrelevant what the number is. What's more relevant is getting alignment.

CIO Insight: Does this begin to address some of the concerns you have about bringing back the human element to looking at value?
Riazi: I'm not certain. I mean, I've heard a lot of great ideas here. We always fall back on our comfort zone, and I think in a debate we fall back to our analytical approach and a logical approach to a problem which I don't find to be logical. I find it to be emotional. I think, going back to your question in terms of ROI, the best way to make it work is to look at an organization organically. And it goes back to what Denise said: You have to look at it strategically, [you have to] look at it organically, and acknowledge [ROI] will help, but it shouldn't drive it. The reason it's failed miserably is because it's been the driver in many ways, and it's been sold as a driver. And that's where it becomes very difficult for it to deliver.
I [will] just close with one story with my nephew. I went to buy him a Halloween costume. He wanted to be Superman, so I bought him a Superman costume-he's eight years old. I took the costume home, and I looked in the back, and it said, "This does not enable you to fly." (laughter) And that's an important lesson for us. The message out there is this technology, all this wonderful stuff is going to make you fly. It doesn't, and it won't. And somehow we need to come to terms with that. I'm hoping you guys find the answer because all the CIOs are waiting for that answer, and the CFOs who are dishing out a lot of money.

Hirji: On that point, if anyone argues that technology confers competitive advantage, I think they're deluded. It's not the technology, it's the people.

Riazi: That's all of us. I mean, the majority, has been sold [on IT] this way that it's going to make you fly.

What I would argue is: If we have the best technologists, they can't help but deploy the technology better, cheaper, faster, smarter than anybody else. And so the ultimate focus here is get the best people in place, they'll take care of it after that. So I agree, but I think the challenges are about building those kinds of cultures, building those kinds of organizations.

The challenge, I think, is a business challenge, it's not ours as CIOs. The business has to say I own this and this is my problem. Don't go putting in a big ERP and see me a year later. Alignment is not only IT to business, it's [also] business-to-business alignment, because