Two years ago, a report by the EPA on data center energy use was scathing. In 133 pages, it detailed—often with a note of alarm—foreseeable trends in the nation’s energy-gobbling data centers, and the future was not bright. In response to the EPA’s “call to arms,” all manner of human energy and ingenuity has been mustered to formulate a sustainable, and environmentally protective, game plan for information technology (IT) energy management.
To the IT industry’s credit, it has indeed responded to what has been (justifiably) called its “energy profligacy,” with remarkable speed and impressive results. In just over two years, a virtual paradigm-shift has taken place in the data center industry—redesigned, revamped, and renovated data centers now operate at greater efficiency, and their energy use is measured and managed while the internal constituent hardware and software has been redesigned for energy optimization. And all of this was instituted and well coordinated in a very short span, providing a model to be emulated by other industries facing similar energy and environmental challenges.
In the recent past, energy efficiency simply wasn’t an issue for IT. After all, data centers were the darlings of global high-tech, driving and energizing a new epoch: So who would dare suggest they be put on any stingy electricity budget? And given Moore’s Law’s of exponentially increasing computing strength, IT could easily defend its consumption of just about any amount of electricity because in many ways no other industry has used power so efficiently by delivering so many benefits-per-kilowatt. With end-users demanding server power, the focus was never on efficiency but, rather, uptime reliability.
Then came some disturbing discoveries detailed by the EPA: Even as computer usability and efficiency had grown exponentially, so too had their power consumption.
A few notable stats—which have widely circulated:
- From 2000 to 2005 in the US (and worldwide), IT energy consumption doubled. (Stanford University and LBNL, 2007)
- IT in 2005 consumed all the energy equivalent to 5-gigawatt (GW)-size power plants, or all the color TVs in the US, or nearly 6 million households.
- The total cost of IT that year came to $2.7 billion in the US, and $7.2 billion worldwide.
- Just a year later, in 2006, another source calculated that IT, combined with its extensive networking infrastructure, consumed about 1.5% of all US electricity, at a cost of $4.5 billion. Another study suggested that total cost, with cooling plant costs added, raised the figure several times higher.
- More recent data (2009) from McKinsey & Company/Uptime Institute shows data center energy consumption, in the top third of sites researched, grew 20–30% annually in 2006 and 2007. This rate far exceeded the EPA’s prediction of a 9% growth from 2006 to 2010.
Worst Yet to Come?
Maybe more disturbing are the imminent trends EPA and others foresee:
- If unabated, data center energy demand will double energy consumption from 2006 to 2011 (i.e., to 7.4 GW in 2011).
- Peak-power demand will soar to 12 GW (compared to 7 GW in 2007).
- The McKinsey/Uptime report envisions even higher figures—a need for up to 10 more power plants by next year, and 20 more by 2015 (total, 30) (“Revolutionizing Data Center Energy Efficiency,” April 30 2009, presented at the Institute Symposium on Green Enterprise Computing).
Given such remarkable figures, EPA also challenged the industry to come up with a coordinated response and to collaborate on developing better practices and technologies. Success would mean that, instead of a demand curve shooting ever higher, it could actually decline—saving, in one scenario, $1.5 billion in power costs almost immediately, and, in a few years’ time, several billion more, not to mention dozens of metric tons of greenhouse gases.
What’s Been the Industry Response?
K.C. Mares of Megawatt Consulting, who specialized in energy-efficient data center design and engineering, reports “a tremendous increase in not only products but in discussion” on how to shrink IT’s gigawatt appetite. He notes that the locally based, widely respected Silicon Valley Leadership Group held its first summit on the subject in 2008, and repeated with a second this year. It culminated in displays and demonstrations of hundreds of new products, services, and technologies.
“This in itself was somewhat novel, because the data center industry as a whole has been somewhat secretive about how each company does business, how they operate their centers, and what they do in them,” he says. “So here we are, for the first time, having a whole bunch of end-users, including many that compete against each other, showcasing what they’re doing to increase energy efficiency in their data centers, collaborating with others about how to increase energy efficiency, and then spreading the gospel, if you will, about it. It’s a dramatic improvement and change over where the industry was.”
Perhaps the root of the historical problem and challenge being faced, he continues, is that most data center servers need to run just about all the time—but actual computing occurs during only a fraction of that time. In practice, “they’re, generally on all the time”—burning electricity—and “generating heat in rooms that need to be kept cool.” 
Serving “Free” Cooling
By far, the greatest energy-saving opportunity comes from a simple two-part concept.
1. Separate the hot and cold air flows. Everyone knows computers make heat that’s blown out by exhaust fans at 90˚F or more. It must then be neutralized in the room by inlet cooling. In less efficiency-minded times, that heat was allowed to mix freely with incoming cold air—an extremely inefficient arrangement.
So, the logical answer: Why not instead vent the warm air to the outdoors so that the incoming coolness doesn’t need to be quite so chilled? It’s called cold aisle/warm aisle containment, and it has quickly become the industry’s design standard for new and remodeled data centers.
Ralph Renne, who is director of site operations for NetApp (Sunnyvale, CA)—a major supplier of data center storage solutions, which adopted a policy of freely sharing its energy expertise with the industry and with its clients—explains: “If you don’t have a mixing of the air … you’re then able to deliver the cold air to the locations where it is required, and you eliminate a ‘short circuiting’ or a blow-by effect,” that occurs when warm and cold mix.
In hot aisle/cold aisle arrangements, the fronts of the server racks (where cool air enters) face each other, and the backs, all emitting warm air, likewise. To keep air in the respective aisles from commingling, Renne just suspends a vinyl curtain at the ceiling, “off of fusible links far below the sprinkler systems.” If a fire alarm should ever trigger, “the vinyl containment system will drop out of place, and sprinkler heads can work, according to their original design,” he says.
 |
Photo: NetApp
Airside economizing can mitigate cooling costs. |
Hence, there’s little or no retrofitting of the sprinkler system, and remodeling work is minimal.
However, some HVAC airflow considerations do come into play, as respective cool and warm fan pressures must be kept in balance. Conceptually, its not unlike inflating a balloon with cool air, using the HVAC system, while at the same time, the exhaust fans are “filling a warm aisle balloon,” so to speak, with heat. The two “balloons” must “inflate” at about the same rate.
“We adopted a cold aisle strategy,” continues Renne, meaning that it turns out to be easier to balance the respective fan energy requirements by focusing a cool air inflow set point.
2. Use “airside economizing” a.k.a “free cooling.” Instead of relying on notoriously energy-intensive air-conditioning plants, bring in cool air from outdoors.
“If a facility can retrofit to utilize outside air, by all means we would recommend that they do so,” advises Renne.
This idea assumes the site is not located, say, in a hot desert or tropical zone. NetApp’s Bay Area locale enjoys a mild climate “that allows us to operate in a free-cooling mode,” he says—with outdoor temperatures in the 60s or below, 5,000 to 6,000 hours annually (i.e., much more than half the time).
“We’re able to operate in essentially a completely ‘free-cooling’ mode,” meaning that the mechanical plant, chilled water pumps, chillers, cooling tower, condenser water pumps, and cooling tower fans “are all shut down. We’re just relying on the outside air to cool our data center; that provides a tremendous operating efficiency.”
Apart from the power to run the servers themselves, all that’s needed are the modest kilowatts to run the lighting and air-handler fans. Taking the two methods together, “existing data centers could implement and realize … in many cases, tremendous energy savings,” he says.
Another efficiency complement is to downsize to a more efficient chilled-water plant. In this particular site, a variable-flow primary chiller allows operation at 0.49 kW per ton, “where the benchmark generally is about 0.65 kW per ton.” The smaller unit is a big efficiency improvement.
Of course, far better is to bring in the coolness of the great outdoors.
Savings in this makeover at NetApp’s Engineering Data Center were calculated by local utility Pacific Gas & Electric at “close to 10 million kWh annually saved, through the energy efficiency measures just on the mechanical side,” he reports. For this achievement NetApp was awarded a spectacular $1,427,000 payout from PG&E.
Renne estimates simple project payback will come in five to six years. Thereafter, “over life of facility, we’ll realize tremendous savings.”
A second NetApp data center housing about 300 racks’ worth of servers, consuming about 680 kW of load, received a similar cold-aisle containment strategy retrofit in 2007; wireless sensors were also installed to help monitor and control conditions. The $167,000 investment is realizing $105,000 in yearly savings; payback should come in just 18 months.
Mares’ firm also routinely assists data centers in designing cold aisle containment, and likewise advises installing variable-speed fans, compressors and pumps.
Regarding air-economizers, Mares also notes that a certain former caution in the industry, fearing damp outdoor air effect on sensitive circuit board, has lessened. “Most servers are now capable of running at humidity ranges between 20% and 80%,” he notes, compared to a range of just one-fourth that a few years ago, and can withstand 80˚F, much warmer than earlier specs allowed. Incoming air thus does not need to be nearly so chilled. Moreover, not only can cool air be economized; so can the chilled water. These days at his clients’ sites, he says, “We’re essentially just using cooling tower water to cool the air to servers, instead of using chillers as well.”
At one recently remodeled site, “we’re able to use water economization—just cooling tower water, meaning no chillers for all—for 60 hours of the year.”
Chilled water plants can thus be sized much smaller, and run less often. Coming someday, he predicts, refrigerant will even be incorporated directly into servers themselves, eliminating the need for buildingwide cooling—at least for the equipment.
Revamping Electrical Systems
Besides dealing with cooling inefficiencies, the IT industry has tackled electrical systems head-on as well. Data center electricity uses three elements—power supplies, UPS backup systems, and transformers—and hence there are three areas on which to focus efforts against losses.
Regarding transformers, there’s a move to replace 480-volt output systems formerly typical in data centers, with the 208-volt commercial distribution standard. “Lower voltage transformers are very efficient—on order of 98%,” notes Mares—compared to higher-voltage systems, with efficiency closer to 90%.
The payback on this change can come in “as short as a year—which is surprising, since it is a fairly costly item,” he adds, but the impact multiplies across the whole data center.
Another improvement underway is in the rack-level power supply. Typical racks holds up to 40 servers each; previously, every server would have its own dedicate power-supply, often with two or three, for reliability; total, up to 120 power supplies per rack. Power-supplies usually operate at less than 50% rated capacity; hence, the total inefficiencies “add up to significant losses,” notes Mares.
One good solution—though still cutting-edge—has been to integrate high-efficiency power supplies into the racks themselves. Attached transformers then reduce the voltage for each of the forty or so servers. This arrangement is proving to be “much more efficient, as there’s not as much waste of all these redundant power supplies,” says Mares.
The power supplies themselves (AC to DC transformers) are also being made more energy-conscious. Half a dozen years ago, their efficiency was only about 60%, and the rest was wastage heat. “Now 80% is the standard, and industry leaders are buying even more efficient transformers above 90%,” says Mares.
Even batteries—which don’t consume much energy—have at least one small contribution to make, as manufactures are striving to reduce the considerable floor space they occupy. Enersys has begun offering a 16-volt high-capacity front-terminated battery, which delivers equivalent power, occupying about 20% less real estate, and thus, less space to cool.
More Hard and Soft Solutions
Other notable measures range from altered server architecture to power management and even organizational restructuring. Mares describes several key ones:
Power self-managing software. Programs that once ran continuously for the sake of convenience and uptime, now are instructed to reduce power or shut down when servers aren’t in use. Servers are really only doing tasks about 10–15% of the time, Mares points out. This adds up to a lot of waste. Newer programs can measure aggregate loads and then power-down server elements that are not needed (or power-up quickly, on demand). However, there remains a certain reluctance to go this far, due to reliability issues.
“Virtualization” and real-time data compression. These buzz terms essentially mean that more server loads can be packed into fewer hardware systems. Improved real-time data compression delivers; for example, storage of 100 terabytes (TB) of databases packed into a 10-TB physical disk, eliminating that much server load.
More frequent hardware changes pay off. Instead of upgrading every three years or so as before, buying new equipment every 18 to 24 months enables sites to take advantage of “the new breed of servers that are ... significantly more energy-efficient,” says Mares. Though they draw more power, their performance is even greater. Payback is quick, “because, typically, more is spent on electricity to power servers than on the server itself,” he notes. So, the real savings comes in reducing aggregate power consumption.
IT interfacing with facility management. In the past, IT administration and facility management were often functionally a bit removed from each other. Now “reintroducing” them to each other, and teaching them to work more cooperatively, results in companies being likelier to save money in overall operational cost. This “is really a new trend,” Mares is finding. “Most large organizations are either actually consolidating the two into one, or making sure the two are talking together regularly.”
Power metering and monitoring. Simply knowing energy costs enables better energy management and cost containment. Formerly, meters typically just measured a facility’s power in the aggregate. Now, by adding considerably greater refinement and detail, managers can discover all kinds of interactions and interrelationships. Understanding how the servers and the facility systems often “play against each other—or with each other—helps to reduce energy use,” says Mares.
This relationship has multiplied the application and use of spot metering, enabling energy usage to be better monitored, measured, and tracked. Increasingly, centralized control of power strips and server activity is also possible. “Though not terribly new,” he says, the technology has been “increasing in sophistication and capability dramatically” in the last few years.
Accountability: Metrics and Recognition
Another area where the data industry has stepped up smartly is in overall site energy quantification, benchmarking, and, even, accountability. A few highlights:
“PUE” and “DCiE” Metrics. Responding to EPA’s specific request to come up with ways to measure site energy efficiency, the IT industry devised what it calls the “power usage effectiveness” index, or PUE. Expressed as a simple ratio, the top figure measures total power usage at a data center, and the bottom is the portion used for data processing. The higher the ratio, the less efficient the site; a theoretical “perfection” in energy efficiency would be a ratio approaching one.
A second ratio was also devised, being the reciprocal number, called the Data Center Infrastructure Efficiency, or DCiE. When the PUE first came forth a couple of years ago, an industry average quickly emerged: Most data sites came out at about PUE 2—meaning that for every watt needed to do computing, another was used to support the facility (i.e., to do lighting, heating, cooling), yielding a total of 2 watts over one.
Mares comments: “That means you’re consuming the same amount of energy in the infrastructure as the IT load itself uses; that is not very efficient.”
At any rate, the concept itself caught on everywhere, “And now we’re seeing … keen competition as companies are touting their better PUEs than the next one,” he says. But he explains: “It’s not so much, ‘mine’s better than yours; I get a prize,’ but, ‘I’m achieving a PUE that shows my data center’s efficient. And here’s what I’m doing to achieve that.’
“For the first time, we’re actually giving awards … to operators and owners [who] are operating more efficiently. And the metrics are there to help users not only to measure but also to compare their energy efficiency of that of others.”
Organizations like the EPA, large California utilities, and IT industry monitoring groups are all catching on to PUE/DCiE indexing and keeping tabs on who’s doing how well, he notes.
As for real improvements: Thanks to rapid and intensive efforts, average PUEs have already fallen in half, to about 1.5, in just two years. And the latest, fully optimized data centers are scoring 1.2 to 1.3. One cutting-edge operation has cracked the decimal barrier and hit a point below 1.1.
“So, we’re now less than 10% of the IT load being used to cool and power it,” notes Mares. “This is mind-boggling efficiency.”
GREEN Building LEED. Still another quickly developed and impressive break-through has been the industry’s effort to integrate data center issues into the well-established LEED standard for site efficiency. The US Green Building Council (USGBC) oversees LEED.
The push to incorporate data centers brings a certain irony. USGBC’s longstanding rating system has heavily emphasized earlier-generation “green” concerns, like recycled materials and bicycle racks—yet have all-but-ignored electrical efficiency, which is now belatedly recognized as by far a site’s biggest footprint.
So, in mid-2008, a lobbying effort prodded USGBC to revise its LEED points to accommodate data center energy achievements. A new standard was rushed into the pipeline last autumn; USGBC wanted to mull it a year or two, but the IT industry applied real clout, and the change now looks good to happen by year’s end.
Very soon, watch as data centers change their market boast from “highest uptime,” to “lowest PUE, and greenest power.”