The Wild, Wild Cost Of Data CentersBy Eric Chabrow | Posted 05-12-2008
The Wild, Wild Cost Of Data Centers
Kenneth Brill sees what many others don't: an ominous financial crisis in the making for business, stemming from energy waste in the corporate data center. Like Al Gore, who travels the world warning of an environmental catastrophe caused by global warning, Brill evangelizes on a more-targeted aspect of the environmental crisis, warning IT and business managers about the fiscal and operational consequences of the inefficient energy use of IT.
As the founder and executive director of the Uptime Institute, a member-supported data-center advisory service, Brill warns company executives that they must change the way they finance and manage data centers and their components (servers, blades, etc.) in order to save millions--and perhaps tens or hundreds of millions--of dollars.
This crisis is being fueled by the increasing divergence between rapid performance gains in server computing and far slower advances in energy efficiency. Brill characterizes this as a meltdown of Moore's Law, while others see it as Moore's Law run amok.
At the root of the problem is simple math. Server compute performance has been increasing by a factor of three every two years, but energy efficiency is only doubling in the same period. That widening gap means that the cost to power the data center will increase significantly year after year. And that will cost business big time.
Brill recently spoke with CIO Insight Editor Eric R. Chabrow. What follows is an edited transcript of that conversation.
CIO Insight: You've been warning about a coming crisis in the data center. How bad is it?
Kenneth Brill: It's actually the IT story of the year: The success of Moore's Law has come to a point where we've reached system limitations. The rate of computing has increased two to three times every 24 months, and the weight of the hardware is such that buildings have a difficult time supporting it.
Power consumption is so significant that accommodating the increase in IT power consumption over the next five years will require us to construct 10 major power plants. If current trends continue, we will need another 20 between 2010 and 2015. That's 30 power plants that need to be built just to accommodate the growth in IT power consumption.
Why is this happening?
Brill: When you buy IT hardware, a certain number of watts are going to be consumed when you plug it in. The ratio of embedded watts per $1,000 of hardware has risen dramatically, so even if you spend the same amount of money on IT hardware, that new hardware's increased performance brings a vast increase in power consumption. This happens invisibly for most organizations, because the power bill doesn't go to IT.
Are you suggesting it's coming out of nowhere?
Brill: People at the working level, such as the members of our Uptime Institute, have seen a 6 percent to 7 percent compound annual growth rate in power consumption in their data centers. This is across five million square feet of computer room space, so it's a pretty big sample. Although 6 percent to 7 percent doesn't sound like a lot, over five years, it increases your power consumption by almost 50 percent.
Something very dramatic happened at the end of 2005. We don't know what caused it, but it's abnormal, and there's no historical precedent for it. Power consumption had been going along at a 6 percent to 7 percent compound annual growth rate, but for the top third of our members, the growth rate went up to 25 percent annualized. That means, in two years the power consumption of the top third of our members went up almost 50 percent! There's no historical precedent for that.
Do you know why this happened?
Brill: No, but it looks as if it's recurring in 2008. One of the reasons may be that we're adding new servers at a rate of around 16 percent. If you don't uninstall the old ones, your consumption will go up by the rate at which you're adding servers.
The biggest opportunity for savings is by uninstalling the equipment that has become technologically obsolete. Up to 30 percent of the servers in a typical data center are no longer in active production, but they're still consuming energy. They need to be turned off, removed from the racks and disposed of.
Is that just poor planning on the part of IT organizations?
Brill: Unfortunately, efficiency is not cool. Unless senior executives demand the ability to uninstall hardware, the tendency will be to leave the old stuff installed.
Would turning PCs and other equipment off at night save a lot of energy?
Brill: Absolutely. There are 160 work hours in a month, but in actual hours, there are almost three times that, 24 hours times 30. By turning equipment off when it's not needed, you can reduce power consumption by that enormous ratio.
We need to become more energy-conscious, and we can do it either for a green reason or for profitability. A single server costing around $2,500 consumes almost $750 in electricity per year.
In the Northeast, that could be significantly higher, though in the Northwest, where they're getting power from a hydro-dam at three cents per kilowatt hour, it would be less. That's a lot of money to have something running when it's not doing anything, especially when it cost only $2,500 to begin with.
Have any data centers run out of power?
Brill: In our survey, we found that our members are, on average, about 56 percent utilized--that is, the load as a ratio of their capacity is about 56 percent. But many members that are growing at a 25 percent-plus annual compound rate are up in the 80 percent to 90 percent range, so they've got to build more data-center capacity.
In fact, the biggest boom in data-center construction is going on right now. Many people may think this is a one-time catch-up effort, but I don't believe that's true. Unless the fundamentals change, every major company will need to build a new data center every several years.
Facility costs are currently running at 8 percent of IT's budget. The IT budget is growing at the rate of about 6 percent per year, and the facility's expenses are growing at 16 percent a year. So there's a fundamental incompatibility because the growth in facilities expenses will crowd out other IT initiatives.
That also changes the way you have to budget for IT. Investing more in capital expenditures means less money is available for application development.
Brill: Yes, and I don't think that's good. We don't want to see all this money going into facilities, because it doesn't add to the productivity of IT. What does add to the productivity of IT is application development. If this course were to continue, there would have to be other cuts elsewhere in IT to make room for the growth of facilities and their cost. This is a senior-executive issue.
What's the impact of this data-center crisis on the role of the CIO?
Brill: The skill set of the CIO is changing, because the issues I'm talking about are financial: accounting, capital expenditures, budgeting. This is a skill set that needs to be developed.
People think that a $2,500 server is so cheap you don't need to worry about it. But over three years, the cost of electricity--$750 in a good location--nearly equals the cost of the server, and that's without the CapEx for building the data center or the cost of running it. The idea that servers are cheap so we don't need to be very careful about managing them does not reflect economic reality.
Building good financial models to determine the true total cost of ownership has to be a priority. These costs must be built into application justification decisions, but today they typically aren't.
A classic example is a $22 million investment in blade servers made by an IT organization without consulting the facilities group. Later, the facilities group said, "We've got to spend $50 million to add power and cooling capacity, and another $30 million to operate the plant we've built for the blade servers." So it turned out to be a $100 million decision.
These are career-limiting decisions, and they are not uncommon.
Is something more fundamental going on?
Brill: There's a convergence of several things: The number of servers being purchased is increasing annually, and consumption per server is going up, as is the price of kilowatt hours.
Do you think it's controllable?
Brill: In the short term, we can harvest a lot of gold just by being smart, using technologies like virtualization, for example. One of our members in Europe virtualized its 3,150 servers, ending up with 150 servers. They're saving $2.1 million worth of electricity a year and have recovered $22 million worth of facility capacity.
Haven't you suggested that virtualization is a one-time saving?
Brill: Yes. And when we're justifying new servers, we need to look at what the total cost is. That's the long-term issue.
Why not have some kind of system of governance in a company to control these things?
Brill: The energy consumption of a data center is 20 to 40 times that of an office building. In a data center, the biggest potential for improvement is the efficiency of the IT hardware. Beyond turning off some of it, buying more efficient IT hardware would help.
Manufacturers offer efficient hardware, but it costs more. And most organizations--judging by their buying patterns--look only at the original cost. It's smarter to spend more money up-front on more-efficient hardware, so you don't have to build a data center. It also reopens the discussion about whether and when to use mainframes, midrange computers or servers, because those platforms have different energy efficiencies.
A mainframe consumes 18 kilowatts of power. If it were a server, it would consume 400 kilowatts of power. A mainframe is very highly utilized, and servers are poorly utilized. And the power consumption per transaction on a mainframe is less than it would be on a server--even a virtualized server.
So that means re-examining how you develop applications and systems.
Brill: This gets back to basic strategy issues about what platforms to run applications on. Historically, facility costs were not considered in these decisions. Because of the change in energy consumption per dollar of spend, all the old rules need to be re-examined. People talk about the power crisis or the heat-density crisis, but fundamentally, it's an economic crisis.
Why do you think people haven't realized that?
Brill: I don't know. All the big 10 financial services companies have half-billion to billion-dollar data-center construction programs. These costs are going to hit their profit-and-loss statements. When they do, it will change the economics of IT.
People think that this half-billion dollar to billion-dollar construction program is a one-time catch-up event, so they're budgeting for it that way. I'm saying that they will be repeating it again within four to five years--and again four to five years after that. If they adopted some of the things we're talking about, that would extend the length of time between construction cycles--maybe stretch it out to eight years.
How many organizations have shifted the budgeting of data-center costs to IT?
Brill: Morgan Stanley made the shift recently. It's still rare, but it's beginning to happen.
Does the government have a role in this, and are they doing what they should be doing?
Brill: The Environmental Protection Agency, the European Union and the Chinese equivalent of our EPA are all talking to one another. Historically, IT's energy consumption was below their radar screen. Now, since data-center consumption as a percentage of total consumption has been rising at a 100 percent rate every five years and is projected to continue, it has reached their radar screen.
Government has been trying to educate and establish measurements. Energy Star will come out later this year with an Energy Star rating for servers, which will allow buyers to understand the energy efficiency of various servers--and there are significant differences.
One of our member companies, which buys 100,000 workstations a year, looked at the energy efficiency of the power supplies included in those workstations. They bought from a vendor that didn't even advertise that its power supply was efficient. And the payback was in less than nine months. Since the life of the asset was 36 months, it was a very good investment.
Should government be more proactive in establishing rules governing power usage in data centers?
Brill: My fear is that they're going to get forced into it and that they will not be very subtle. The dynamics of our economy are such that whatever is true today will be obsolete within three to five years, but regulations tend to get cast in concrete and are very difficult to change. I don't see the need for regulation because the economics are so compelling: It's in people's own best interests to be more energy efficient.
Would it be in the best interests of the vendors to manufacture more energy-efficient IT?
Brill: Absolutely. They've done this, but those products are languishing in their inventory because users don't realize it's in their best interests to buy more energy-efficient products.
How about the impact of rising fuel costs?
Brill: Most electricity is produced on long-term contracts that won't be affected by the rise in fuel prices for many years. Even when those fuel prices catch up, they won't have much effect because fuel is a small percentage of the cost of producing electricity. Most of the cost of producing electricity is in the CapEx investment: the power plant, along with the transmission and distribution lines. Even so, it's prudent business to eliminate waste.
We need to throttle back the demand for IT by having the true cost of ownership reflected in the cost of applications. It's in our own best interests to know what's paying off and what's not.