Welch Foods: Out of a Jam with Virtualization

By David F. Carr

Welch Foods: Out of a Jam with Virtualization

As Welch Foods was shutting down a grape processing plant in Kennewick, Wash., and transferring operations to nearby Grandview this winter, managers at the Grandview plant asked the information-technology staff to speed up the replacement of one of the outdated servers from Kennewick. In years past, fulfilling that request would have meant ordering a server, waiting a couple of weeks for it to arrive, configuring the machine, installing the application and shipping it cross-country along with a technician to set it up. "It would have been three weeks, minimum," says Jacob Matusevich, manager of networks and server technologies, in a mid-March interview at his office in Concord, Mass. Instead, he got it done in less than an hour.

How? Instead of deploying a new physical server, Matusevich's staff was able to activate a new virtual machine on a virtual server—ESX Server from Palo Alto, Calif.-based VMware, running on Dell hardware and deployed in Grandview as part of the consolidation of the Washington State plants. Even though computer operations employees manage the West Coast plant remotely from Lawton, Mich., and from headquarters in Concord, Mass., they can now respond to these requests quickly—often within minutes—marking a significant change in the way Welch's runs its computers.

Server virtualization makes it possible to host multiple applications on a single physical server, but within an isolated software environment that lets each application run as if it were on its own. That means each application can run on a different operating system or version of an operating system, or the same one with different configuration tweaks. This simulated computing environment is known as a virtual machine.

With VMware, each of these virtual machines can be reduced to an image file—a snapshot of the server environment, including the application, operating system and configuration choices—and that file can be moved or copied from one machine to another. And that's how Welch's can bring a new server online quickly, sometimes in as little as 15 to 20 minutes, by cloning a previously prepared server image, customizing it as necessary and setting it to run on a VMware host.

Welch's is the food processing and marketing arm of The National Grape Cooperative Association; it makes and sells Welch's juices and jellies and distributes the proceeds to the growers, who are essentially its shareholders. Welch's is headquartered in Concord, Mass., and has production facilities in New York, Pennsylvania, Michigan and Washington. Its growers manage nearly 50,000 acres of vineyards in Pennsylvania, Michigan, New York, Ohio, Washington and Ontario, Canada, and harvested 413,344 tons of Concord and Niagara grapes in 2005. But in 2005, the company was also recovering from a poor 2004 harvest, which conspired with rising expenses to cut profits to $58.5 million from $75 million the year before. Revenues in 2005 remained flat at $577.8 million, up by $292,000 over 2004.

The closing of the Kennewick plant is one symptom of the cost-cutting Welch's has engaged in to improve its margins.

On the scale of Welch's operations, the direct cost savings from virtualization are small, although "every little bit helps," Matusevich says. He estimates the savings from running 10 ESX servers in place of about 100 standalone servers at $331,000. That is, each ESX server costs about $38,900, including the Dell server hardware and four-processor VMware license, or $389,000 for the 10 virtual servers. To achieve comparable functionality by purchasing standalone servers, Welch's would have had to spend about $7,200 each, or $720,000 for 100 units.

But the use of VMware, which Welch's began researching in 2002 and first deployed in 2003, was driven less by cost savings than by the flexibility it offered for day-to-day operations, backup and recovery, and disaster recovery, according to Matusevich.

For day-to-day operations, Welch's currently runs 14 ESX servers, including the 10 at headquarters. Deployment of VMware servers to grape processing plants around the country is just beginning, spurred by the Washington plant consolidation. There has to be a physical server equipped with VMware at a location before a virtual machine can be deployed.

As a result, Grandview was the first plant to deploy an ESX server early this year, following the decision to close the Kennewick plant. Given the need to consolidate computer operations between the two plants and scrap some outdated server hardware, George Scangas, Welch's senior infrastructure analyst, recommended consolidation of about a dozen servers onto a single Dell server with redundant hardware, running the ESX software. The remote management virtues of VMware were important to this decision because Grandview has no I.T. staff of its own, instead getting support from Rick Reeves, a technology manager based at a plant in Lawton, Mich.

This was Reeves' first exposure to the VMware system, which is now also being deployed in Lawton. Scangas provided training and built the first couple of virtual machines, then supervised while Reeves built the rest. "I'd read the literature, but it was even easier than I'd expected it to be," Reeves says. "You don't often find that." He is now following up with a VMware deployment at his own plant.

ZIFFPAGE TITLESchooling the Old School

Schooling the Old School

Virtualization of Intel-based computers is becoming increasingly common with the maturation of products from VMware, the virtualization market leader, and competing products including Microsoft's Virtual Server. But Scangas, who has the most hands-on responsibility for Welch's VMware environment, says he still runs up against "old school thinking" from managers who feel more secure running their applications on a separate server.

The idea that each application needs to run on its own server may seem alien to desktop computer users, who have grown used to running many programs at the same time in different windows of a multitasking operating system. But one application per server has become the default deployment for Windows and Linux servers in many data centers. That simplifies life for data center operators by keeping a crash in one application from shutting down others on the same machine, while allowing operating system parameters to be tuned for the best performance of that one application. But this setup can also be very wasteful, with many servers running at 5% or 10% of their processor and memory capacity. Isolation of applications on one computer within a virtual machine or partition has been available for years on mainframes and high-end Unix systems, but is relatively new on Intel-based servers.

Not every application lends itself to this treatment. The Oracle enterprise resource planning system that Welch's is in the midst of deploying will run on physically separate servers, for example, because the software is so demanding in terms of database access and processor utilization. But several other Oracle database applications run on virtual machines and do just fine. Applications that have worked well on virtual machines at Welch's include the Cognos data analysis and reporting server, Microsoft SQL Server, Microsoft Exchange and the webMethods integration server, as well as domain controllers, File Transfer Protocol servers and other basic network services.

Scangas says it's not really the importance of the application that makes him hesitate to put it on a virtual machine. In fact, several of Welch's most critical systems, including databases that manage its juice formulas and grower records, run in the virtual environment. Rather, it's the behavior of the software itself that has to be evaluated before it is loaded onto a virtual machine.

For example, software with heavy input/output (I/O) requirements may not run well because it is constantly issuing read and write instructions, which aren't as efficient in a virtual machine environment. Instead of talking directly to the device driver for a storage system, for example, the software communicates with a simulated representation of the device, and the virtual server software redirects that request to the actual hardware. For most applications, the added delay is insignificant. But when Welch's tried to run sales forecasting software from Demantra, a planning software vendor, on VMware, the virtual machine was quickly overloaded.

"The box just wasn't keeping up," Scangas says. The Demantra application never came close to exhausting the memory he had allocated to it, but the heavily customized analytic model Welch's had created required too many read and write commands for VMware to handle, he explains. The application has since been moved onto its own IBM pSeries server.

Jerry Normandin, a Welch's Linux specialist, says he has learned to create a profile of processor use and other resource demands before virtualizing an application. The types of applications that work well are "not ERP, but smaller systems," he says. "If an application doesn't use a lot of I/O, it's a good candidate for a virtual machine. If it's not a heavily threaded Java application, if it's something with light to intermediate use, it's a good candidate."

Welch's originally deployed VMware's GSX Server, which uses Windows as the base operating system on which the virtual machines run. But the company quickly graduated to ESX Server, which it judged to be better for enterprise-scale deployment. ESX uses as its host environment a slimmed-down, proprietary operating system—essentially a thin layer that contains the device driver, juggles the other "guest" operating systems and incorporates elements of Linux. With either the ESX or GSX products, the guest operating systems within the virtual machines can be Linux, Windows or one of several others that run on Intel processors.

ZIFFPAGE TITLEMore Room in the

Data Center">

More Room in the Data Center

Giving a tour of the data center at Welch's headquarters in Concord, Scangas points to a mostly empty server rack as evidence of what VMware has accomplished for the company. "These racks used to be filled, but we've retired a lot of the older equipment with virtualization," Scangas says. Instead of replacing each of the old servers with a new one, Welch's has consolidated the applications that used to run on those old servers onto 10 Dell PowerEdge 6650 and 6600 servers running ESX Server. Each ESX Server runs 10 to 13 production applications, on average, or 15 to 18 for development and testing.

"Without VMware, we would not fit in the room right now," agrees Tom Barry, the data center's manager; the room is about 40 by 40 feet. Even with more compact computers such as blade servers, however, power and cooling requirements grow rapidly as more servers are added, he says: "I've been here six years, and we've gone from 35,000 watts to 120,000 watts; if it weren't for VMware we'd probably be at 240,000 watts." Like several of Welch's senior I.T. executives, he readily accepted the idea that virtualization would be an improvement because he had experience working with mainframe virtual machines at previous jobs.

Nelson Arcoraci, group manager for computer operations, said the only question he asked was, "Is it ready?" Once assured that VMware's software was production-quality, he was quick to give the go-ahead. "We have about 65% of our infrastructure running on it today," he says, and more software is likely to move onto virtual machines over time. "For 90% of applications, it's a good fit."

Arcoraci and Matusevich both say the ability to use VMware for disaster recovery was another major reason for pursuing the virtualization strategy. "Intel infrastructure can be very picky with regard to recovery," Arcoraci says, because when a server is restored from backup tapes onto another computer, the hardware configuration on that machine may not be what the software has been configured to expect. But if the software is running in a virtual environment, it can be restored onto another machine without any dependencies on the underlying hardware.

For backup purposes, the virtual server strategy also helps limit the number of servers that Welch's has reserved for emergency use at the IBM data center in Sterling Forest, N.Y. "Instead of asking IBM to have 100 servers for us, now it's just 10," Scangas says, which cuts the cost of backup service. Even some of the applications that normally run on separate servers could be consolidated onto virtual machines in an emergency, he adds.

Virtualization has helped Welch Foods sweeten its bottom line and boost efficiency—and, perhaps most important, kept its technicians off those coast-to-coast flights.

ZIFFPAGE TITLEWelch Foods Base Case

Welch Foods Base Case

Headquarters: 575 Virginia Rd., Concord, MA 01742
Phone: (978) 371-1000
Business: As the food processing and marketing arm of The National Grape Cooperative Association, Welch makes and sells juices and jellies.
Chief Information Officer: Larry Rencken
Financials in 2005: Sales of $577.8 million for the year ended Aug. 31, up by $400,000 from previous year. Net proceeds were $58.5 million, down by $16.5 million.
Challenge: Use virtualization technology to contain data center costs, while simplifying server deployment and administration.

  • Reduce the cost of procuring, configuring and supporting servers by up to 30%.
  • Save about $331,000 in server hardware costs. Also, cut data center costs by reducing space and cooling requirements and trimming electric power consumption.
  • Cut the time to deploy a server from as many as five weeks to as little as 15 minutes.
  • Achieve average server process utilization of about 60%, compared to about 10% for physically separate servers.


: Saving Time Through Virtualization">
Welch Foods: Saving Time Through Virtualization

Scenario: Welch Foods, based in Concord, Mass., receives a request to deploy a new but fairly standard server to one of its remote locations, such as its Grandview, Wash., plant. Here is how the company potentially saves time by using server virtualization.

Without Virtualization Time With Virtualization Time
Procure the server. 2 to 4 weeks Copy complete image of software environment,both application and operating system, onto remote VMware server. Tweak configuration as necessary and go live with application. Less than 1 hour
Configure operating system. Install and configure applications. 4 to 6 hours Not necessary
Ship server cross-country. 1 to 3 days Not necessary Not applicable
TOTAL TIME 16 to 32 days TOTAL TIME Less than 1 hour

This article was originally published on 04-06-2006