Building Empirical Metrics
Gauging return on investment for e-business applications presents a Gordian knot: the benefits are the new processes they bring, yet because the processes are new, existing measurement tools may miss those benefits.
What is the best way to measure the effectiveness of an e-business application and its value to a company? Ask the people most affected: the users themselves. Develop and implement a well-structured survey that asks all the end-users how their jobs are changing as a result of the application. Doing so offers the benefit of metrics that are based on real and measurable impact. It measures change where it matters most.
This empirical approach provides a flexible method of developing useful ROI data. First, it is possible to design survey instruments in such a manner that one can probe specifically for certain benefits and improvements you expect from the application. You can also do hypothesis testing for changes and benefits that are hard to find by any other measurement. For example, some customer-relationship management implementations have lowered turnover in customer-support staffa real cost savingsbut that would not have been evident without interviewing the people affected.
The empirical approach takes the established practice of IT customer surveys to a whole new level. There are a number of important differences between it and traditional surveys. First, the survey has to be given regularly and on a known schedule to produce the metrics needed. Second, it must have a consistent questionnaire and track the same people. Third, it should focus as much as possible on quantifiable items to curtail subjective responses. Remember, just as with metrics generated by systems, consistency of process and data gathering is critical.
One important aspect of the empirical approach is that it is possible to gather a baseline set of data before you install the application. IT, finance and line-of-business representatives can set their expectations as part of the design process. This approach allows you to use the same metrics for measuring the return from actual use you developed in the planning phase.
Looking to the user community in a structured, documented and consistent manner may often be the only way to build accurate metrics for many e-business applications. The benefits of finding the hard, soft and unexpected value points of these new applications is critical to meeting ROI goals.
Aaron Goldberg, a member of Ziff Davis Market Experts, concentrates on strategic marketing issues. Comments on this story can be sent to firstname.lastname@example.org.
This article was originally published on 05-01-2001