The escalating costs for power and cooling in data centers are two of the biggest issues facing companies today.

By Lyn Corum

 A series of reports started appearing in 2006, most notably from the Lawrence Berkeley National Laboratories (LBNL), documenting the inefficiencies and costs in operating data centers, sometimes as high as 40 times that of the typical office complex.

Press releases from information technology (IT) companies in the past year have announced new software and hardware solutions for these escalating costs. Power management, energy-efficient building design, and virtualization are the new catch phrases for the latest developments in cutting power costs in data centers. This new information, plus conversations with representatives of companies, industry groups, and research institutions, identifies much that can be done by IT managers and owners of data centers without waiting for the results of current research.

In a 2007 report to Congress, the EPA claims that the nation’s servers and data centers spent $4.5 billion for electricity in 2006—that’s 1.5% of total US electricity consumption (more than double the electricity consumed by servers and data centers in 2000)—and is expected to nearly double again by 2011, from 61 billion kWh to more than 100 billion kWh.

Servers and data centers are creating peak loads in the power grid, estimated to be approximately 7 gigawatts—equivalent to the output of about 15 baseload power plants. The EPA estimates that, if current trends continue, this demand will rise to 12 gigawatts by 2011, requiring an additional 10 power plants.

Even the Department of Energy’s (DOE) Energy Star program has come onboard, including power management as one of its products. Its Web site provides guidance on reducing the energy used by computer equipment (www.energystar.gov).

Measuring Energy Use
The Green Grid was founded in 2007 as a global consortium of companies dedicated to advancing energy efficiency in data centers and business-computing ecosystems. The nonprofit trade organization released its first white paper, Green Grid Metrics: Describing Datacenter Power Efficiency, written by a technical committee made up of members soon after it was founded. Its founding members and board of directors represent AMD, APC, Dell, Hewlett-Packard (HP), IBM, Intel, Microsoft, Rackable Systems, SprayCool, Sun Microsystems, and Vmware. There are now 139 members.

According to that first white paper, growing companies need a way to control data center costs while expanding and growing. “With more efficient data centers, IT organizations can better manage increased computing, network, and storage demands, lower energy costs, and reduce total cost of ownership . . . while remaining competitive and able to meet future business needs,” it says.

The Green Grid developed a set of metrics, which it continues to refine, to help IT organizations better understand and improve the energy efficiency of existing data centers. Two metrics, power-usage effectiveness (PUE) and, its reciprocal, data center efficiency (DCE) were created:

PUE = total facility power ÷ IT equipment power
and
DCE = IT equipment power ÷ total facility power

IT equipment power includes the load associated with all the IT equipment in the data center, as well as supplemental equipment. Total facility power includes everything that supports the IT equipment load, such as power delivery components, cooling system components, computation, network and storage notes, decreased efficiency of UPS equipment, and miscellaneous loads such as data center lighting.

For example, if a data center’s PUE is calculated to be 3.0, it means the demand is three times greater than the energy necessary to power the IT equipment. It can be used as a multiplier for calculating the real impact of the system’s power demands. If a server demands 500 watts, then it needs 1,500 watts from the utility grid to deliver that demand.

On the other hand, a DCE value of 0.33 (the reciprocal of the PUE), suggests that the IT equipment consumes 33% of the power in the data center.

The Green Grid authors of this white paper soon realized there are real-world complexities that make these metrics simplistic. They pointed out that a mixed-use building may house various data centers, labs, or offices, making the determination of the power usage of one data center very difficult, particularly if the utility power enters the building through a single entrance point in the utility room. Furthermore, the latest cooling technologies integrate such cooling elements as pumps, refrigeration, blowers, and heat exchangers within the IT equipment itself.

In October 2007, Green Grid released another white paper, improving on the data center metric. The Green Grid Data Center Power Efficiency Metrics: PUE and DCiE (data center infrastructure efficiency) redefined DCE, in which:

DCiE = 1 ÷ PUE = (useful work ÷ total facility power) X 100

Total facility power is defined as the power measured at the utility meter dedicated solely to the data center. It will include UPS, power distribution units, distribution losses, cool system components, and so forth. If the utility meter measures power drawn from the utility for an entire office building, the power used by the non-data center offices will have to be measured or estimated and subtracted out. As with DCE above, a DCiE value of 33% suggests that the IT equipment consumes 33% of the power in the data center.

The authors point out that there is currently no comprehensive industry data set that shows accurate PUE statistics for data centers, and there is no general agreement on what constitutes an efficient or inefficient data center. Ideally, a PUE value approaching 1.0 would indicate 100% efficiency. The paper quotes a 2006 LBNL study, discussed below, reporting that the 22 data centers measured had PUE values in the 1.3–3.0 range. It also says that other research indicates that PUE values of 2.0 are achievable with proper design.

The paper recommends that IT managers begin measuring data center efficiency, even if the method currently requires data manipulations, and to share and compare respective PUE or DCiE results with other managers and data center owners. The Green Grid promises eventually to offer values that profile target PUE and DCiE metrics for a variety of typical data center configurations.

The Green Grid also promises to work with the industry to define energy efficiency guidelines for all of the components in the data center: the uninterruptible power supplies, switchgear, chillers, computer room air conditioners, direct-expansion units, pumps, cooling towers, generators, distribution losses external to the racks, power distribution units, batters, lighting, servers, and storage.

The Green Grid is working on metrics to define data center productivity—useful work as a function of total facility power. Data center productivity is harder to determine, but members believe it is a key strategic focus for the industry.

On a separate note, the Green Grid announced in September 2007 it had signed a memorandum of understanding with the US DOE Office of Energy Efficiency and Renewable Energy. It will collaborate on educating IT managers on how they can improve the energy efficiency of data centers and the associated financial benefits. It will also develop a common set of best practices information, terminology, and tools, to promote increased energy efficiency in data centers.

The Green Grid simultaneously announced the creation of two working groups that will focus on data collection, technical analysis, and communications in European data centers.

In a more recent development, the Green Grid technical committee that developed the white papers presented a new framework at a Green Grid technical forum in February, addressing organizational barriers to managing efficiency. It also discussed a baseline efficiency market study that reviewed the current state of the industry, identifying the key factors driving companies to take action on data center power consumption, as well as the challenges involved. For more information, readers should go to www.thegreengrid.org and select “News.” Also, the white papers can be found under “Content.”

Transforming the Data Center
What are some solutions? Mark Monroe, director of sustainable computing at Sun Microsystems, represents the company on the Green Grid board of directors. In an interview, he describes five ways to increase the energy efficiency of data centers and cut costs by reducing power demand, shrinking the footprint of data centers, and increasing the efficiency of IT equipment. The five ways are: pure technology refreshment, consolidation, virtualization, storage area networks (SAN), and data center assessment.

Technology Refresh
Equipment three to five years old takes up more space and consumes more power, says Monroe. He recommends replacing old computers with new ones before other actions are taken.

To measure this dynamic, Sun created an innovative metric called SWaP—a space, watts, and performance metric allowing computer users to determine the impact of a server in the data center. It is calculated as:

SWaP = performance ÷ (space X power consumption)

Using that formula, one can determine what the most efficient equipment would be. For example, Server A, doing 500 operations in space measured as 2 rack units (RU) using 300 watts, has a SWaP rating of 0.83. Server B, producing equal performance in a larger space, uses more than twice the power—performing 500 operations in space measured as 4 RU, it uses 800 watts, and has a SWaP rating of 0.16. Therefore, Server A is five times more efficient than Server B. The reader can calculate his own SWaP metric by going to www.sun.com/servers/coolthreads/swap/index.jsp.

Monroe described what happened when Sun replaced all the equipment in its data center in Holland with new equipment. Power consumption was reduced 75%, rack space was reduced 80%, and cooling requirements were reduced 75%.

Consolidation
Second, a data center is made more efficient through consolidation and can be carried out in tandem with technology refreshment. Consolidation may require changing or upgrading an operating system. At Sun’s Santa Clara campus, the company used compression to consolidate hardware, swapping out 2,200 older machines for 1,000 new machines. Storage boxes were cut from 750 to 230. In the process, end users were able to achieve 2.5 times more computing jobs, and 4.5 times more storage capacity.

The new computing equipment cost $7 million, but Sun saved $9 million on construction costs since it could build a smaller facility. Power to generate cooling was reduced from 2.2 MW to 560 kW, saving $1.5 million annually.

Virtualization
Always use virtualization in tandem with consolidation, cautions Monroe. Virtualization can be accomplished at several levels—with software and with the physical separation of hardware. In traditional computing, the application is installed in a desktop. In virtualized space, there is a three-tier architecture that makes hardware appear to be a number of virtual machines, says Monroe. Applications are installed in servers, and an infrastructure tier in the middle delivers applications—such as a Sun Ray device that can run any operating system—to the client’s desktop.

The Sun xVM hypervisor virtualizes a system’s hardware. It transparently shares and partitions the system’s CPUs, memory, and other resources throughout the user domain.  It is a software layer that sits between hardware and the operating system, and isolates applications. Operators don’t know other users are sharing the system. Consequently, space, power, and cooling are reduced. It also reduces the administrative burden by cutting back on the number of machines needed. Other virtualization technologies include the Microsoft Virtual Server. 

Monroe used an example. Application developers like to think they have control of an entire machine, but sometimes they make mistakes that won’t allow different applications to work together. With hypervisor software in place, the application assumes it has the whole machine to itself. But in fact, Solaris-, Linux-, and Windows-operating systems are operating on the one machine and a programming mistake in which one operating system cannot migrate to another. In the past, three machines—one for each operating system—would have been required.

Photo: Sun Microsystems

Sunlight deflectors

There is also storage virtualization that can enhance service levels and remove physical limitations. Such storage modules as Sun’s Solaris container are located in the physical infrastructure, taking up the disk capacity. Above that are the rapid application development (RAD) engines that deliver hardware protection levels and performance characteristics. Virtual disks are created using a virtualization engine and can be remotely located from the disks, controllers, or connectivity.

This virtualization offers several advantages. In the desktop space, security is at the forefront, where everything must be protected. Centralization of the applications does this. Disaster recovery plans and regulatory compliance work can be stored miles away from a site. Storage virtualization can also optimize the tracking of what users are using and help employees work remotely.

Storage Area Networks
By replacing the need for another machine, SANs offer another solution for growing storage requirements. SAN can be described as a pool of storage that many servers can access. This application may be of special interest to small- and medium-sized businesses that traditionally buy software off the shelf. When energy costs were not important, the businesses could buy all the hardware needed, but no more. SAN makes it cost-effective to virtualize and consolidate storage for applications from various sources such as Sun, HP, IBM, Linux, and others, without fear that these will interfere with each other. This means the applications can be run on fewer machines.

Data Center Assessment
Finally, Monroe discussed a suite of programs that Sun offers for data center assessment, also known as retrocommissioning, to make sure each building is operating at its optimum. This is where the metrics developed by the Green Grid can provide useful information. Using those metrics, IT managers can calculate how efficient or inefficient the data center is, then move on to the Sun programs, which will assess, optimize, and virtualize the data center, starting with the electrical and mechanical systems, and tuning up the equipment to such design standards as resetting air temperature and plugging holes.

Monroe says just over half of energy used goes to operating computers, according to a LBNL study. Cooling energy use can be cut by 15%, the study showed, producing an eight-month payback. Other studies have found computers suck up anywhere between 30% and 70% of the electricity used in a building.

Industry Helps Itself
The Silicon Valley Leadership Group (SVLG), made up of over 200 Silicon Valley member companies that lobby local, state, and federal government, created an energy-efficient data center demonstration project by gathering together a group of end users with data centers. These were matched with vendors and providers like Sun Microsystems, Cassatt Corp., Intel, and Bay Area universities, plus others. All worked together to develop examples of smart, energy-efficient data center computer equipment, operations, and supporting infrastructure.

A spokesman for SVLG, Frank Teng, says the purpose was to see what the real-world experience is with the technologies being developed by the vendors and providers. By end users doing a “proof of concept” test, it gives new technologies and strategies a record, helping to proliferate more rapidly, reports Teng.

Teng says 20 case studies will be completed by this summer and will be introduced at SVLG’s Energy Summit on June 1. Following the summit, a report will be available at SVLG’s Web site (www.svlg.net). The LBNL and the University of California Davis Efficiency Center supported the demonstration project.

LBNL on Cutting Edge
In earlier research, the LBNL studied 22 data centers, developing a set of best practices to improve energy efficiency in data centers. These included improving air management by emphasizing control and isolation of hot and cold airstreams, right-sizing central plants and ventilation systems to operate efficiently, improving configuration and operation of uninterruptible power supplies, high-efficiency computer power supplies to reduce load at the racks, and direct liquid cooling of racks or computers.

Photo: Sun Microsystems

Cooling unit

Just as importantly, the authors recommended onsite generation combined with absorption chillers for cooling using the waste heat, ideally with grid interconnection to allow power sales to the utility.

For additional best practices recommendations, one can read the paper published in 2006 on LBNL’s Web site at http://eetd.lbl.gov/emills/PUBS/PDF/ACEEE-datacenters.pdf.
LBNL led another demonstration project with a group of 85 companies, including Sun Microsystems and Intel, with funding provided by the California Energy Commission’s Public Interest Energy Research program. This was to test the notion that direct-current (DC) voltage would greatly reduce energy consumption in data centers. A number of reports and videos on its Web site explain the test and document the test installation (http://hightech.lbl.gov).

Bill Tschudi, the principal investigator, says results of the study indicated that the DC approach does provide an increase in conversion efficiency. He says the consensus was good on 380-V DC use. Efficiency gains of 5% to 7% were found in high-efficiency UPSs and power supplies. However, a typical alternating-current (AC) system in today’s data center, the study noted, would have a UPS that is 85% efficient—down from the 90% high-efficiency model—and power supplies around 73% efficient. Estimated improvement of a DC system over the typical AC system was estimated at 28%, and this improvement would extend to the overall facility, the report says.

Tschudi says switching to DC systems will have to jump the classic commercialization hurdle. Manufacturers do not yet make the hardware, such as power strips, and are saying there are no buyers, in spite of the existence of early adopters. The LBNL is trying to keep the momentum going and continues to receive funding from the Public Interest Energy Research (PIER) program. The DOE came calling in early March and may add to the PIER funding, says Tschudi.

What Else Is Industry Doing?
Since the EPA report was released in August 2007, software and hardware companies such as HP, Unisys, The Planet, and Cassatt Corp. have announced equipment and software innovations that promise to substantially reduce energy use in data centers.

HP announced in January a new product line of energy-efficient, business desktop PCs.  The dc7800 Ultra Slim Desktop is small, with a reported 85% energy-efficient power consumption and solid-state hard drive, offering improved reliability, according to HP’s published statement. The dc5800 Business Desktop PC, in a redesigned small form factor, also has Intel processors to help reduce processor heat and energy costs.

Unisys Corp. opened an outsourcing IT data services facility in Eagan, MN, in December 2007, featuring an energy-efficient building design, glycol air conditioning with enhanced efficiency, a monitoring and control system that allows power usage to be continually balanced with fluctuating heating and cooling requirements, and virtualization technology.

In February, The Planet, a provider of on-demand IT infrastructure services hosting more than 22,000 small- and medium-sized businesses and millions of Web sites, announced the results of a six-month trial. It was able to reduce power consumption by 31% and increase data center operating efficiency to 1.7, from the previous “good” ranking of 2.0.  This is an EPA-recognized measurement of the total power necessary to operate a data center divided by the critical power required to operate its computers.

The Planet was able to reduce power costs at its six data centers, by optimizing the efficiency of its computer room air-conditioning units and increasing air-conditioning unit temperature settings by as much as 10˚F. The changes included extending return air plenums on all of its down-flow air-conditioning units. It also rearranged floor tiles to better manage cold airflow; installed seals and grommets in the ceilings, walls, and floors to reduce bypass airflow; installed blanking plates in server cabinets to direct airflow more efficiently; and sealed power-distribution units to reduce bypass airflow.

Cassatt Corp., which creates technologies and software solutions for data centers to improve resource, operational, and energy efficiencies, announced its Active Power Management tool in September 2007. The tool works with existing resources in a data center to determine which servers to power down and for how long, and then handles application requirements before turning the servers off. As servers are needed again, it applies the same logic and priorities to power the servers back on. It runs on any platform.  

California-based writer Lyn Corum specializes in energy topics.

DE - July/August 2008