How the Customer Satisfaction Survey Was Done

By Allan Alter  |  Posted 12-15-2005
What does the survey measure? CIO Insight's 2005 Vendor Value Survey measures how U.S.-based IT executives generally perceive the value of their vendors' product and service offerings, and those executives' overall satisfaction with the support these vendors provide.

How were the vendors selected? The published results include only vendors that received 40 or more qualified responses on all ratings, and were used by at least 60 respondents. To create our list of the most widely used U.S. computer hardware, software, telecommunications, IT consulting and outsourcing companies, we relied on several sources for revenue and market share data: Gartner (which provided CIO Insight with lists of market share leaders, the Fortune 1000 list, the Forbes Private 500 list, the Software Magazine Software 500 list, Consultant News, Hoover's Online and annual and

financial reports of individual companies. Most of the companies in the 2004 Vendor Value survey were included this year's survey; deletions were due to insufficient unqualified responses. Our aim was to provide readers with information on approximately 40 of the most widely used vendors in these categories; at the end, results on 41 vendors were published, out of 53 in the survey. We limited hardware companies to manufacturers of PCs and servers, storage and networking equipment, and software companies to producers that create and sell software intended primarily for business use across many industries.

Want to see how your vendors did in last year's survey? Click here: Are Your Vendors Providing More Value?

We included companies that provide consulting, systems integration, outsourcing and IT services (other than data processing services and business process outsourcing services), which do not derive most of their IT service revenues from the defense sector, information processing services or customers outside the U.S. We also included the six largest U.S. telecommunications services vendors, as ranked by the Fortune 500, but did not include companies that are primarily wireless telecom or cable service providers.

How was the survey conducted? CIO Insight editors designed the 2005 Vendor Value and Satisfaction Survey together with Equation Research, LLC (www.equationresearch.com), an Estes Park, Colo.-based supplier of custom research services. IT executives gathered from Ziff Davis Media publication lists were invited to participate in the study by e-mail. The questions were posted on a password-protected Web site, and 884 qualified respondents (459 from companies with between $5 million and $1 billion in revenues, and 425 from companies with over $1 billion in revenues) replied from October 4 to October 24, 2005. Of the respondents, 57 percent were the top IT executives at their companies, and the rest held titles of director of IT or higher. Respondents were only considered qualified if they described themselves as very knowledgeable or knowledgeable about the IT vendors and consultants their company uses, and the value it has received from them.

How were vendors rated? After identifying the vendors they have had a business relationship with in the past 12 months, respondents were asked to rate vendors as "excellent," "good," "fair" and "poor" on seven key criteria. Four of the criteria concerned value: 1) how well they have met their company's expectations for increasing revenues (or achieving mission, if not-for-profit), and 2) for lowering business or IT costs; 3) how well they have solved the business problem their products or services were purchased or engaged to solve, and 4) have met their company's ROI (business value) expectations. The other three criteria focussed on reliability: 5) how well they met commitments to their company on time and on budget 6) how flexible and responsive they have been to the company's needs, and 7) how well they met the company's quality expectations for their products and services. The "overall" rating is the mean of respondents answering "excellent" or "good" for these seven criteria. In addition, respondents were asked whether they would continue to do business with each individual vendor. Unless otherwise noted, percentages given are the percentage of respondents who answered either "excellent" or "good."

The exact wording of the survey questions is as follows:

Please indicate which of the following vendors your company has had or continues to have an ongoing business relationship with during the last twelve months

Please indicate the nature of the relationship your company has (or had) with ______ in the last 12 months.
· We have used them as consultants on business and IT strategy and/or internal IT projects
· We have used their services to outsource IT systems and projects
. We have used their telecommunications services.
· We have used their hardware products
· We have used their software products

Please rate _____ on how well their products and services help your company achieve the following goals and expectations:

Meeting my company's expectations for increasing revenues (or achieving mission, if not-for-profit)
· Meeting my company's expectations for lowering business or IT costs
· Solving the business problem their products or services were purchased or engaged to solve
· Meeting my company's ROI (business value) expectations
· Meeting commitments to my company on time and budget
· Being flexible and responsive to my company's needs
· Meeting my company's quality expectations for their products and services

If you had a choice, would you continue to do business with ______?

Survey Guide:
Are Your Vendors Fighting to Keep You Happy?

  • CIOs Describe Like-Hate Relationships With Vendors
  • Winners and Losers: Survey Shows Shifts in Customer Esteem
  • Excelling in Spots: Customer Evaluations by Category
  • How The Survey Was Done

    Want to see how your vendors did in last year's survey? Click here: Are Your Vendors Providing More Value?