Building Empirical Metrics

By Peggy Sue Heath

Trends: eBusiness Strategy



In a world where companies are rushing to harness their operations to the cost savings and speed of the Internet, a strategic e-business system can have a big impact on how you're doing competitively in the marketplace.

Just ask Ricoh Corp., the West Caldwell, N.J.-based copier company. Because it had a customer relationship management system that could help it identify its customers' specific needs, it was able to snare a $200 million, 10-year contract to be the sole provider of office automation equipment to the U.S. Postal Service. "E-business is critical to being a player in our industry," says Joseph Vallorosi, vice president of IT at Ricoh.

But what separates Ricoh from many other companies is that it realizes it's no longer enough to have e-business systems in place. You've also got to know their precise impact on the company's bottom line—and the potential dollar pay-off to your customers, old and new: Ricoh's winning bid convinced the Postal Service it would save $7.4 million a year using Ricoh's products. Says Vallorosi: "Knowing the impact of e-business to the customer's bottom line was the critical difference between us and our rivals."

Sounds like common sense, but the truth is that not many CIOs get it yet.

And no wonder. Up until now, the rush to harness operations to the Net has left CIOs little time to analyze the business need for e-business systems before they install them. Further, traditional methods of calculating return on investment have been, for the most part, ill-suited to measuring the strategic impact of e-business on a company's bottom line. Such old methods rarely showed the application's true value to the business. Indeed, say e-business experts, very few companies have properly measured the impact of their e-business systems, either before, during or after installing them.

CRM is a good example. According to Stamford, Conn.-based research firm Gartner, Inc., through 2004, only 35 percent of companies planning to install a CRM system will adequately develop the tools they need to evaluate their new system's potential business pay-off. And of those, less than 20 percent will use such tools to evaluate the success of the installation over time.

Is ROI dead? Not at all. It's simply changing to meet the sobering new demands of the digital economy. Indeed, the tightening economic climate is helping to fuel a new emphasis on ROI. "Regardless of industry, people are asking for more rigorous cases that illustrate both the short-term impact and the long-term benefits of a potential implementation," says Randy Hancock, senior vice president of strategy at Mainspring Inc., a Cambridge, Mass.-based strategy consulting firm.

The goal: to develop an ROI approach that not only provides an initial justification for new e-business systems but also establishes a baseline for their ongoing management. Ideally, such an approach will measure the potential value of the proposed system against the business goals of the company.

"You have to assess both the opportunities for increased revenue and the minimized costs of new processes in the company," says Ricoh's Vallorosi. "Once you go live, you've got to ascertain the cost savings and make sure the increase in revenue is measured and attributable."


-Business Value">

E-Business Value

Determining the value of e-business is not about isolating the benefits of a single application. It is about assessing the impact of the Net on the overall business once it has been transformed through e-business.

Traditionally, ROI has been measured with metrics such as Gartner's Total Cost Ownership, which is designed to reveal a project's impact on productivity. But measuring the ROI of an e-business application is more challenging. Harnessing your operations to the Net can critically affect strategy—beyond the improvement of processes and productivity. For example, e-business can fundamentally change the customer experience, a company's products and even its business opportunities.

Last year, North Vancounver, B.C.-based North Shore Credit Union, with $820 million in assets, made a large financial bet on CRM, with an implementation from Pivotal, Inc., a Canadian software company. The hope was to increase assets, customer loyalty and the productivity of its account managers.

"We are an extremely targeted community provider, and to continue to do that effectively, we have to manage carefully our customer relationships," says Chris Cutliff, North Shore's president and CEO. Among North Shore's goals: to generate a digital data profile of every customer within a year, and an even more detailed one for its high-profile customers.

Says Elaine McHarg, the former senior vice president of marketing at North Shore and the business owner of the project: "We needed to attract younger customers who were earlier in the financial life cycle, and we wanted to be able to give them more targeted advice, based on their individual financial profiles. We needed to secure more of the share of those high-profit, high-profile customers. And we needed to manage that across all of our touch points and products."

The firm was able to leverage its popular online "financial wellness" check—a Web-based data profiling survey—into a tool for collecting more detailed customer data for use by marketing and sales staffs.

And the ROI: The CRM project has boosted the productivity of North Shore's customer relationship managers significantly. "The preliminary numbers show the account revenue from our financial managers is up 18 percent after five months," says Cutliff. "Our loans have grown 20 percent on an annualized basis, and our retention rate has gone from 74 percent to over 90 percent."

Using the old way of measuring ROI, North Shore would have overlooked a majority of the CRM system's potential and real benefits—including the bank's ability to attract an entirely new type of customer. Instead, they would have focused on comparing the costs of implementing and managing the new CRM system with the old way of doing things. The result: the CRM project may not have gotten funded at all.

The advantage of the new ROI metrics is that they view the IT department as a value center. The new metrics (see Building Metrics) focus on the value a particular IT investment can or will bring to a variety of yardsticks the company considers vital: the price of a company's stock, its corporate revenues, its profits, costs and customer relationships, business process improvements, cycle times and productivity—even the potential for corporate growth over time.

Each new way to measure ROI has a different focus and usefulness, but choosing the right one requires CIOs to ask a couple of key questions. First, should the new ROI method be used to supply the data needed to justify a one-time cost, or should it also provide CIOs with the tools needed to manage the entire e-business system? Some ROI tools, for example, are best suited to helping CIOs justify their one-time costs, select vendors and develop a business case for an e-business technology that links it to company strategy. But CIOs may also need to think further ahead.

"The companies with dramatic success are those that don't see the ROI effort as purely for justification and then leave it behind," argues David Kohar, director of e-strategy for Pivotal. "Instead, they align the proposed investment with company goals and then use that case to guide implementation and drive the initiative with clear metrics, goal posts and strategy."

Choosing Methods

Choosing Methods

Many managers want their ROI efforts to lead to metrics that help them track, reward and motivate staff. Consider methods such as the eStrategy Balanced Scorecard (from Mainspring and the Balanced Scorecard Collaborative, Inc., a Lincoln, Mass.-based consultancy), which starts with justification of the initial project but is designed to remain useful throughout the life of the project by providing metrics and incentives for managing the implementation. The method begins by identifying the desired economic results of a new e-business strategy and then focuses on creating the activities necessary to achieve those results, says Mainspring's Hancock.

The second key question: Do you prefer the view through the windshield or the rear-view mirror? Valuation methods tend to be either forward-looking or retrospective. Economic Value Added (from Stern Stewart & Co.), Economic Value Sourced (from Meta Group) and The Customer Index (from Accenture) can be used to pinpoint the value of an investment at a fixed moment in time, based on financial performance over a specific period. As such, these methods can be very useful for conducting regular reviews of past performance and making decisions about enhancements.

To be meaningful, however, these tools require actual numbers, not projections. This makes them more difficult to use as part of building the initial ROI justification for an individual e-business project.

Indeed, some of these methods are limited to situations in which a new e-business system is designed to enhance a company's existing strategies. Once they're implemented, the value of such investments can be more easily isolated.

But when a company is investing in e-business in hopes of radically reforming the company and its products through innovation, "managers quickly realize that traditional tools don't help that much because they are fixated on predictable and obvious outcomes," says Adam Borison, a partner at PricewaterhouseCoopers in Menlo Park, Calif.

PricewaterhouseCoopers' Real Option Valuation method is designed to go beyond the tangible near-term results and put a value on future possibilities. The approach—derived from financial techniques for valuing options contracts—puts a value on the flexibility of the system, the options it opens up for the company and management's ability to leverage those options.

The advantage of all of these new methods over the old ones is that they try to capture the often intangible, "soft" benefits of an implementation. Each approach has its limits, but they all can provide insights into the impact of a company's bottom line. Regardless of the method used, the key for CIOs is to begin with the goal in mind—the new system's desired impact on the company and its revenues and costs.

Building Metrics

Building Metrics

Each of these valuation methods assumes that the data needed to generate metrics is being gathered and that some baseline knowledge of the current system is available. Baseline metrics are needed both to support the initial ROI justification for a proposed system and to allow for its ongoing evaluation.

In the case of an e-business implementation that will affect customer support, for instance, metrics assessing the customer experience need to be established before the system is put in place. Without setting a baseline that includes such criteria as customer satisfaction, customer retention rates and new-customer adoption rates, for example, it will be difficult to set realistic expectations for a new system—much less gauge the actual benefits.

"It was important for us to have clear expectations going in," says Liz Lloyd, manager of information systems for North Shore. "We brought in an outside firm and spent a couple of months working to understand the scope of what had to be done before we chose a vendor. This first round of analysis helped produce reasonable estimates rather than guesses."

Establishing metrics at North Shore, however, required a lot of estimation and validation of data that didn't previously exist. At North Shore, says McHarg, now a partner with M3, a consultancy in Vancouver, B.C., "We had to take the statistics deeper into the organization than we had originally anticipated. But that reality check with salespeople, relationship managers and others was critical."

McHarg is a strong advocate of metrics that let project owners set precise implementation targets. "When you get going on an e-business implementation, suddenly everything competes for importance," she says. "Building out the baseline and benchmark targets with Pivotal beforehand was invaluable to me in highlighting those key opportunities and focusing our development."

E-business systems can often provide the data that is needed to measure its impact on the company. Analytics that cut across CRM, sales-force automation and e-marketing systems can help track the metrics that matter, such as costs to acquire a customer and average revenue per customer.

It used to be that the customer's experience, for instance, could only be captured through a survey or by aggregating such basic statistics as call-center queue times. An advanced CRM system with analytical capabilities can provide detailed information on individual customers, including their use of customer service channels, and which parts of the Web site they use, and how. In the process, larger patterns in sales, usage and satisfaction will emerge. While the primary purpose of such tools is to help department and line-of-business managers make sales and marketing decisions, it can also help evaluate the e-business investment's contribution to business performance.

Christine Cournoyer, CIO and senior vice president of global e-business transformation at Lotus Development Corp., a division of IBM, focuses on what she calls process metrics and results metrics when evaluating e-business systems. The revenue Lotus derives from e-business is a result metric. But it can be just as important to measure the value of the processes that led to that improvement, such as the number of hits to a Web marketing campaign, how many and which pages were viewed, how many visits generated leads and how many led to sales.

The first step is to understand the e-business project's business objectives and use ROI approaches that help build justification. Then, establish process and outcome metrics for identifying whether those objectives are being met. The ultimate goal is to leverage that e-business investment into something that fundamentally changes the company's ability to compete and succeed.

An Active Role

An Active Role

One consequence of the shift from a focus on IT costs to its impact on overall business strategy is that the role of the CIO is evolving to include some responsibility for companywide metrics and performance. At the end of the day, the job of the CIO, like every other senior manager, is to increase revenues, decrease costs and maximize the market value of the company. So CIOs must assume a more active role in measuring value companywide in order to successfully show return on investment from their e-business systems.

Before buying a new e-business system, the most important jobs of the CIO is to bring financial discipline and accountability to the decision process and to tie the business goals, functional requirements and vendor options together. Even when a project is "owned" by a business line or managed by cross-functional teams in which IT is a participant but not a driving force, the CIO's role is critical.

After the purchase, the CIO and the IT staff must communicate the near-term strategic targets and metrics to be used to measure the project's payoffs. Rather than create new metrics based solely on IT-focused costs and IT-centric benefits, CIOs need to size up the goals of the company and evaluate its business performance.

Understanding the corporate politics of such a shift is important. In many companies, customer-facing e-business applications are heavily influenced, if not owned outright, by sales, marketing or customer support specialists.

"The IT group may be the catalyst, the driver or the enabler of e-business, but we don't do it in a vacuum," says Vallorosi. "Good coordination and cooperation with the business units is important." In some cases, he says, the CIO and the IT team can even become a part of the sales process. "Never before was an internal IT person a part of the customer bid process, but now we are actually participating in presentations to explain our e-business capabilities to prospective customers," says Vallorosi.

Perhaps the most critical responsibility of the CIO is to create an atmosphere that will lead to the creation of new metrics, and ultimately to the success of the e-business system. That's what takes ROI valuations from mere projections to reality. The e-business investment must change behavior in sales, marketing and customer support. In partnership with the rest of senior management, the CIO needs to help educate and motivate employees to make sure the change succeeds. In other words, CIOs play a critical role in making e-business pay off.

Peggy Sue Heath, a vice president of Ziff Davis Market Experts, analyzes emerging technology markets. Comments on this story can be sent to editors@cioinsight.com.

Building Empirical Metrics

Building Empirical Metrics

Gauging return on investment for e-business applications presents a Gordian knot: the benefits are the new processes they bring, yet because the processes are new, existing measurement tools may miss those benefits.

What is the best way to measure the effectiveness of an e-business application and its value to a company? Ask the people most affected: the users themselves. Develop and implement a well-structured survey that asks all the end-users how their jobs are changing as a result of the application. Doing so offers the benefit of metrics that are based on real and measurable impact. It measures change where it matters most.

This empirical approach provides a flexible method of developing useful ROI data. First, it is possible to design survey instruments in such a manner that one can probe specifically for certain benefits and improvements you expect from the application. You can also do hypothesis testing for changes and benefits that are hard to find by any other measurement. For example, some customer-relationship management implementations have lowered turnover in customer-support staff—a real cost savings—but that would not have been evident without interviewing the people affected.

The empirical approach takes the established practice of IT customer surveys to a whole new level. There are a number of important differences between it and traditional surveys. First, the survey has to be given regularly and on a known schedule to produce the metrics needed. Second, it must have a consistent questionnaire and track the same people. Third, it should focus as much as possible on quantifiable items to curtail subjective responses. Remember, just as with metrics generated by systems, consistency of process and data gathering is critical.

One important aspect of the empirical approach is that it is possible to gather a baseline set of data before you install the application. IT, finance and line-of-business representatives can set their expectations as part of the design process. This approach allows you to use the same metrics for measuring the return from actual use you developed in the planning phase.

Looking to the user community in a structured, documented and consistent manner may often be the only way to build accurate metrics for many e-business applications. The benefits of finding the hard, soft and unexpected value points of these new applications is critical to meeting ROI goals.

Aaron Goldberg, a member of Ziff Davis Market Experts, concentrates on strategic marketing issues. Comments on this story can be sent to editors@cioinsight.com.

Measure by Measure

Measure by Measure

Consultants are offering a variety of methods for evaluating the return on investment in e-business systems. Each emphasizes different processes and outcomes within the company.

Economic Value Added

Stern Stewart & Co. measures periodic performance in terms of the change in long-term yield on shareholders' investment, which is defined by the sum total of all revenues minus costs, including cost of capital.

Economic Value Sourced

Meta Group focuses on four areas of value impact for IT investments: increasing revenue, improving productivity, decreasing cycle time and decreasing risk.

eStrategy Balanced Scorecard

Mainspring Inc. and the Balanced Scorecard Cooperative, derived from Kaplan and Norton's Balanced Scorecard, evaluates the value of IT across a scorecard that includes financial indicators, customer perspectives, internal business processes and organizational growth.

The Customer Index

Accenture brings revenue, cost and profit down to a per customer basis in order to evaluate the impact of a particular IT investment.

Real Option Valuation

PricewaterhouseCoopers, based on merger and acquisition methodology, views IT investments as options where the value of the investment is determined by the future range of opportunities the technology represents.

Forget ROI

: Make IT A Profit Center">

Forget ROI: Make IT A Profit Center
Viewpoint by Aaron Goldberg

Are CIOs just defensive by nature?

Consider how much of their time CIOs throughout the IT community spend trying to justify spending. It's all about self-protection and defending costs.

Why not go on the offensive? Now's the time for CIOs to position IT as a profit center that's just as deserving of attention and investment as any other revenue generating department in the company. After all, most new e-business applications are focused on improving operating results through cost reduction and revenue enhancement.

Become more aggressive when positioning IT systems and their value. Traditionally, there has been little impetus to push the numbers so that any financial benefits beyond recovering costs are calculated.

You need to do more than just develop new ROI numbers. Taking IT into profit-center mode requires rethinking some fundamentals. First, decide whether you will move the entire IT function and the services it provides to a profit model or if you will look to specific applications as distinct profit centers.

For those who go for the whole enchilada, the prevalent approach is the "utility" model. Individual users and departments are charged a monthly or annual fee for IT services. Under this plan, IT typically makes a profit by finding internal efficiencies that lower costs, creating a spread between the fees charged and the cost of supplying the services.

I'm not a fan of this approach. There's no "new" value created. The profit is essentially a phantom, driven either by negotiating fees that are advantageous to IT or by walking a tightrope between cutting costs and continuing to provide a reasonable level of service.

I prefer selecting projects that drive real revenue or new cost savings and pushing them into a profit model. Target, for example, a CRM system that measurably increases sales to your existing customers.

Another key benefit: In order to focus on new areas for returns, IT is often forced to align with one or several business units. For example, finding a new way to offer technical support online that reduces costs by 25 percent means that IT and customer service must act in lockstep to determine what the technology can provide, what customers expect, how much change staff can accept and how best to generate a return.

Creating new value, either cost savings or revenue growth, is what real profit centers do, and that attracts top management attention. Just taking current expenses and moving them around among different groups is nothing more than a shell game.




IT Services Costs, Metrics, Benchmarking and Marketing
by Anthony F. Tardugno, Robert E. Matthews and Thomas R. DiPasquale. Prentice-Hall, 2000.

Scaling for E-Business: Technologies, Models, Performance, and Capacity Planning
by Daniel A. Menasce and Virgilio A. F. Almeida. Prentice-Hall, 2000.


Measuring eBusiness Success
by Bobby Cameron with Julie Meringer, Christopher Dawe and Emily Jastrzembski. Forrester Research Inc.
September 2000.

This article was originally published on 05-01-2001