Vendor Value Survey: MethodologyBy CIOinsight | Posted 11-14-2007
What does the survey measure? CIO Insight's 2007 Vendor Value Survey measures how U.S.-based IT executives generally perceive the value of their vendors' product and service offerings, and those executives' overall satisfaction with the support these vendors provide.
How were the vendors selected? The published results include only vendors which received 45 or more qualified responses on all ratings, and were used by at least 50 respondents. To create our list of the most widely used computer hardware, software, telecommunications, IT consulting and outsourcing companies in the U.S., we relied on Baseline magazine's 50 Fastest Growing Software Companies, the Fortune 500 and Global 500 lists, Hoover's Online, and annual and financial reports of individual companies. Most of the companies that were included in the 2006 Vendor Value survey were included this year's survey; usually deletions were due to insufficient or unqualified responses. Our aim was to provide readers with information on the most widely used vendors in these categories, out of 53 that were included in our survey. We limited hardware companies to manufacturers of PCs and servers, storage equipment, and networking equipment; and software companies to producers that create and sell software intended primarily for business use across many industries. We included companies that provide telecommunications, consulting, systems integration, outsourcing and IT services (other than data processing services and business process outsourcing services) but do not derive most of their IT service revenues from the defense sector, information processing services or customers outside the U.S.
How was the survey conducted? CIO Insight editors designed the 2007 Vendor Value and Satisfaction Survey with members of the Ziff Davis Enterprise research staff. IT executives from Ziff Davis Enterprise publication lists were invited to participate in the study by e-mail. The questions were posted on a password-protected Web site, and 472 qualified respondents (170 from companies with between $5 million and $99 million in revenues, 172 between $100 and $999 million in revenues, and 130 from companies with over $1 billion in revenues) replied and completed the survey between September 8 and September 27, 2007. Of the respondents, 51 percent were the top IT executives at their company, and the rest held titles of director of IT or higher. Respondents were only considered qualified if they described themselves as very knowledgeable or knowledgeable about the IT vendors and consultants their company uses, and the value it has received from them.
How are vendors rated? After identifying the vendors they have had a business relationship with in the past 12 months, and whether they use the vendor as a hardware, software, telecommunications, consulting or outsourcing services provider, respondents were asked to rate vendors as "excellent," "good," "fair" and "poor" on seven key criteria. Four of the criteria concerned value: 1) how well they have met their company's expectations for increasing revenues (or achieving mission, if not-for-profit), and 2) for lowering business or IT costs; 3) how well they have solved the business problem their products or services were purchased or engaged to solve, and 4) have met their company's ROI (business value) expectations. The other three criteria focus on reliability: 5) how well they have met commitments to their company on time and budget 6) how flexible and responsive they have been to their company's needs and 7) how well they have met their company's quality expectations for their products and services. The "overall" rating is the mean of respondents answering "excellent" or "good" for these seven criteria. In addition, respondents were asked whether, if they had a choice, they would or would not continue to do business with each individual vendor. Unless otherwise noted, percentages given are the percentage of respondents who answered either "excellent" or "good."
The exact wording of the survey questions is as follows:
Please indicate which of the following vendors your company has had or continues to have an ongoing business relationship with during the last twelve months.
Please indicate the nature of the relationship your company has (or had) with ______ in the last 12 months.
- We have used them as consultants on business and IT strategy and/or internal IT projects
- We have used their services to outsource IT systems and projects
- We have used their telecommunications services.
- We have used their hardware products
- We have used their software products
- We have used their security products and services.
Please rate _____ on how well their products and services help your company achieve the following goals and expectations:
- Meeting my company's expectations for increasing revenues (or achieving mission, if not-for-profit)
- Meeting my company's expectations for lowering business or IT costs
- Solving the business problem their products or services were purchased or engaged to solve
- Meeting my company's ROI (business value) expectations
- Meeting commitments to my company on time and budget
- Being flexible and responsive to my company's needs
- Meeting my company's quality expectations for their products and services
If you had a choice, would you continue to do business with ______?