July-August 2009

Data Centers and DG

IT facilities, which are making great strides in improving energy efficiency, are a natural fit for onsite power generation.

Tweet

Address one environmental challenge and sometimes another materializes. The e-communications revolution that has occurred over the past decade-plus has saved innumerable trees and reduced burdens on landfills and recycling facilities, in addition to exponentially improving the efficiency of information flow. But as technology applications such as Web hosting, financial transactions, and medical record storage continue to increase the need for electronic processing, storage, and networking equipment, energy demand to power it has increased exponentially.

The EPA estimates that data centers consumed 1.5% of the nation’s power, or 61 billion kWh, in 2006, and these facilities’ share will be expected to have roughly tripled by the end of the decade. EPA also estimated that the amount of power consumed by these facilities in 2006 was double the amount in 2000, and that if current efficiency trends continued, total data center power consumption would reach about 100 billion kWh by 2011. The US Congress has taken note: Public Law 109-431 required EPA to conduct a study of the rapid growth of energy consumption in data centers and recommend actions to reduce their energy consumption. An August 2007 EPA report to Congress provided several recommendations for improved operation, best practices, and the state of the art. Some of the recommendations relating to power and cooling include improvements in fans, chillers, and pumps, and the use of free cooling and combined heat and power (CHP).

Several large technology corporations have followed suit and made energy efficiency improvements in their data centers, including Hewlett-Packard, IBM, Google, and Sun Microsystems. Just as significantly, the information technology (IT) industry has developed its own metric for data center power consumption. The Green Grid, an association of IT professionals focusing on the energy efficiency of data centers, proposed the use of Power Usage Effectiveness (PUE), or total power consumption divided by IT equipment power consumption, and the metric has been widely adopted by the IT industry.

Recently, three companies have successfully reduced their data center power consumption using this metric and EPA and industry recommendations. These projects bode well for the future of distributed generation (DG) in general.

Photo: NetApp Interior view of a louver wall at the new NetApp facility, which is used to bring outside air into the facility for cooling.

Sun Microsystems
A recent move by Sun Microsystems to combine two data centers into one in Broomfield, CO, will save the company 1 million kWh of electricity per month and an estimated $1 million in energy costs annually. Design strategies, in the largest data center consolidation project undertaken in the company’s history, included minimizing the raised floor space and optimizing chilled-air distribution. The Santa Clara, CA–based company opened the facility in January 2009 and reduced total floor space by about 75%, eliminated $4 million in costs with the raised-floor reduction, saved more than $1 million in electrical costs, reduced its carbon dioxide (CO2) emissions by 11,000 metric tons per year, and is expected to cut its corporate carbon footprint by 6%.

The facility is the company’s fourth to use a new energy-efficient design, following in the footsteps of data centers in Santa Clara; Blackwater, UK; and Bangalore, India. The Santa Clara facility, which opened in August 2007, was compressed from 200,000 to 80,000 square feet—at a $1-million savings and yielding a PUE of 1.28. Mark Monroe, director of sustainable computing for Sun Microsystems, acknowledges that the company saw an opportunity for cost savings in the wake of the other facilities’ construction and the 2005 acquisition of server storage device provider Storage Technology (StorageTek).

Monroe reports that Sun had spent about $250 million over the previous four years for an initiative to reduce the company’s data center floor space by about one-third. The primary motivation for modifying the acquired StorageTek facility in nearby Louisville, CO, was cost savings, with sustainability emerging in conjunction, Monroe says. “I like to say that we’re pragmatic ecologists, and I think in every for-profit company you’ll find will say the same thing,” he says. “Here in Colorado, we get our power from two coal-fired power plants just down the street. That 11,000 metric-tons [CO2 reduction] is about 5% of our global carbon footprint, and about 6% of our US carbon footprint.”

The Broomfield consolidation is part of a larger Sun initiative to achieve a 60% data center square footage compression globally. This initiative has included a 30% decrease in data center operating expenses in the Bay Area alone since 2007. In October 2008, Sun announced that it had reduced greenhouse gas emissions from US operations by 23%, surpassing its goal five years early; the company is now attempting to reduce its emissions by another 20%, with projects like Broomfield. Using its experience in improving energy in its own data centers, Sun recently launched its own design service for these facilities.

The equipment in an acquired 496,000–square foot lab and data center in Louisville was moved to Sun’s campus in nearby Broomfield, where overall IT equipment floor space was compressed to 126,000 square feet and 165,000 square feet of raised floor was condensed to 700 square feet, says Monroe. This reduction alone reduced much of the facility costs, because it eliminated the need for additional reinforcement to support the weight of the servers, which can weigh as much as 1 ton per rack, according to Monroe. Noting that raised floors have traditionally served dual purposes of facilitating the distribution of both chilled air and cabling, he says, “Unfortunately, those two purposes are at odds with each other because the more cabling you put underneath the floor, the more it restricts the airflow, and the more problems you have with the cooling system because the air becomes unbalanced—you have hot spots.”

The facility’s cooling system uses containment of hot air produced by the servers within the aisles. Another feature of the design, notes Monroe, is the close proximity of the cooling equipment to the IT equipment; in many cases, the hot air only travels 2 or 3 feet before being blown across cooling coils, and gets blown around to the front of the computer. The facility utilizes “close-coupled cooling” with Liebert Corp. overhead cooling systems and APC row-level cooling systems. The design and equipment are much more efficient than a traditional data center design that allows hot air to meander to a register all the way across the room before being blown under a raised floor, Monroe points out.

Additionally, the facility is equipped with two Trane variable-speed centrifugal 500-ton chillers with two-stage compressors that are 25% more efficient than American Society of Heating, Refrigerating, and Air-Conditioning Engineers standards. “We also use quite a bit of evaporative cooling; rather than run the chillers here in Colorado on a day when it’s 45°F and gets down to the twenties at night, we use a flat plate evaporator out in the cooling tower to do the heat exchange, and we can get over 1,000 hours of cooling each year,” says Monroe. The cooling system uses a Clearwater Dolphin closed-loop electromagnetic purification system for the chiller water that saves the facility about 675,000 gallons of water annually, according to Monroe.

Photo: NetApp Powersmiths transformers are being used to increase energy efficiency at a new NetApp Inc. engineering and data center.

The cabling in the facility also has been reconfigured according to Sun’s pod architecture, another design item that reduced costs, Monroe adds. The networking and power distribution cables are located above the server racks and power distribution equipment is now located above the server racks in what is described as a pod design. Power distribution boxes such as three-phase, 208-V or single-phase, 110-V can be twisted into an overhead Universal Electric Corp. Starline Plug-In Busway system, reducing the amount of copper wire needed by about half, according to Monroe. “With a raised floor, no one ever pulls cable out,” he says. “It’s either too long or too much work, so when modifications are made over the life of a data center, copper cable builds up underneath there in the form of networking cables and power cables. With this overhead system, every time we make a modification, we take the old cable out, we put the new cable in, and we’re going to save tons of copper inside the data center, which is a scarce resource these days.”

Another energy saver is the use of flywheel-based UPS rather than traditional lead acid batteries, Monroe points out. “Number one, they run more efficiently all the time, and number two, they don’t have this room full of lead acid batteries that need to be replaced every six years, meaning that you’ve got to dispose of the lead and chemicals,” he says. Monroe recalls a day in February 2009, when high winds caused a power outage but the flywheels started up the backup generators, which provided uninterrupted power for a couple of hours.

Advanced Data Centers
A multidisciplinary team engineering approach to improving energy efficiency resulted in a PUE that is pushing the limit of theoretical achievement for San Francisco, CA-based Advanced Data Centers’ first data center on the former McClellan Air Force Base near Sacramento, CA. The company will provide facility space and equipment for customers that opt to not operate their own data centers. By exhausting equipment-generated heat, making the distribution of chilled water more efficient, and other methods, the team is expected to obtain a PUE of 1.12 when the 200,000–square foot facility is completed in late 2009. The design is estimated to provide about $2 million in energy savings every year, a fact not lost on the Sacramento Municipal Utility District, which awarded the three-year-old startup with its largest-ever rebate under the “Savings by Design” energy efficiency incentive program.

Bob Seese, the company’s chief data center architect, says that the multidisciplinary team approach to facility design that came about in 2007 was a result of the movement toward energy efficiency in data centers.

“At that time, the industry, although it was starting to migrate in that direction, still thought that electric costs and the electricity that was being used by data centers were untouched areas because a data center had to remain operational 24 hours a day, seven days a week, and 365 days a year, and touching energy could, they felt, impact its reliability,” says Seese. “Understanding that that was the industry’s position, I had to support my position. I thought that there was a lot of opportunity for savings in terms of cooling.”

Seese put together a team of experts in multiple engineering disciplines including electrical, mechanical, and structural. Significantly, he adds, collaborations were based on the Rocky Mountain Institute’s charette concept. “We instructed everyone when they came to these charettes to leave their particular specialty at the door and, instead, come into the room and be an engineer that day—not necessarily a structural, civil, or mechanical engineer.”

He credits the approach as instrumental in helping the team to get the facility pre-certified for a US Green Building Council’s Leadership in Energy and Environmental Design Core and Shell Platinum rating.

Photo: NetApp A vari-prime chilled-water distribution system is used in conjunction with these Trane chillers to efficiently cool the NetApp facility.

“They really did find creative solutions instead of just going back to the old habits of doing things the way they had always been done,” says Seese. “A lot of that was based on the naïveté of engineers, who maybe didn’t understand mechanical engineering, but they asked, ‘Why don’t you do it this way?’ And in a lot of cases, the mechanical or electrical engineer would come back and say, ‘Let me see if that solution does work.’”

One of the most important strategies that the team used, Seese points out, was making use of the outside air temperature in the Sacramento area. “We looked at the weather in the Sacramento area and found out that we could use outside air to cool our facility 75% of the year,” he says. “In a typical data center, chillers consume as much as 0.6 kW per ton—that’s a lot of energy. If we could find a way to use outside air and not cause problems with the dust count in the air, we could save a lot of energy.”

Seese adds that Rumsey Engineers of Oakland, CA, which had assisted in the design of many “clean rooms” for semiconductor manufacturing, had successfully utilized outside air for cooling these facilities.

Average high temperatures in the Sacramento area range from 55–82°F between November and May, compared with 89–94°F from June through September. The bigger challenge in bringing outside air into the new facility via high-efficiency fans, Seese says, was controlling the humidity for a few days out of the cooler months—without consuming a lot of power. Noting that the region usually has optimal humidity for the natural cooling strategy, he reports that high-efficiency atomizing dehumidifiers will be used to dehumidify the return air stream.

During months in which it was necessary to chill water for cooling, the facility will use up to six modular centrifugal chillers: five 750-ton units and a 250-ton unit for low-load conditions for initial occupation of the space. The 250-ton unit and one of the 750-ton units are equipped with variable frequency drives; Seese explains that these two units would be used in situations where 850–1,500 tons of chilled water are needed, and the other 750-ton units can be utilized incrementally when more than 1,000 tons are needed. Another significant cost-saving aspect of the facility, Seese adds, is the fact that the chillers are located outside of the facility. “We had to put our chillers outdoors because putting our chillers indoors would have had too much of an impact on the amount of real estate we had to lease out for business purposes—we had a certain number of square feet to rent,” he notes. “They don’t like the rain, so you have to cover them; this solution puts the cooling towers over the top of the chillers.”

“This idea of delivering cold air underneath the floor and expecting that cold air to rise runs counter to the laws of physics,” says Seese of traditional data center cooling system design. He explains that the team focused on minimizing the number of required turns in the distribution of chilled air.

In contrast to a typical data center—where air is delivered beneath a raised floor, makes immediate right-angle turns, and travels across the floor before making another right-angle turn, and finds its way up through perforated floor tiles in front of the computer equipment—the new facility will draw air through the outside air louvers, and the drawn-in air will follow a straight path through filters and cooling coils to the equipment floor. The equipment floor is pressurized so that the temperature of the air from the floor to the ceiling is constant—eliminating the hot and cold air mixing that is found in typical data centers.

Additionally, the team designed a “hot-aisle containment system” that pressurizes air and exhausts equipment-generated heat from the facility to prevent it from mixing with the cooler air. Doors were installed at the end of hot aisles to contain the heat produced by the equipment and blown out the back of the equipment by fans. “Cold aisles” are in the front of the equipment and back up to an enclosure that captures and exhausts the hot air. Roofs were constructed above the servers and chimneys were constructed on top of the roofs. Above the chimneys are plenums with variable frequency drive (VFD)-equipped fans that draw the air in the hot aisles upward and exhaust it.

“At certain times of the year, the [outside] air is too cold, so we allow the hot air to come back in and mix with the cold air coming in from outside to temper it down to 70 to 72 degrees,” says Seese. “The other thing it does is allow us to cool air based on its temperature. The outside air can get to 104 degrees, and our return air could be 95-degree air. It makes more sense for us to cool the 95-degree air, because it requires less energy.

Phot:o: NetApp Exterior view of louver wall used to bring filtered air into the NetApp facility for cooling

“The other thing we’ve done is completely eliminate 90-degree turns in water delivery, and that is obviously done for the purpose of making the water run much more efficiently and smoothly, and reduces our pump energy significantly,” adds Seese.

The resulting higher efficiency, he says, allows the delivery of 55°F water to the chillers, rather than the typical 45°F water. Energy consumption is 0.32–0.35 kW per ton, rather than the typical 0.6 kW per ton. Overall, Seese notes, all of the power consumption, except for 12%, is for the purpose of powering the IT equipment.

The facility also features a UPS that relies on a 3-ton flywheel, rather than Direct Current (DC) batteries for backup power. Seese points out that the flywheel is about 3% more efficient than the best available batteries, which are about 94% efficient, according to a Lawrence Berkeley National Laboratory survey relating to double-conversion UPS systems. The flywheel is more efficient than batteries and allows conversion of alternating current from the grid to DC, exclusively for the data center equipment, Seese says. A Hitec motor/generator that is normally powered by a motor at the local utility would be powered by the flywheel at the data center. “The typical data center uses alternating current, turns it into direct current, turns it back into alternating current, and sends it up to the servers, which immediately turn it into direct current,” says Seese, adding that this occurs to charge the batteries and correct any anomalies in the power that will be delivered to the servers.

In addition to turning the generator, the utility power also spins the 3-ton flywheel, which keeps spinning via its own inertia and continues to spin the generator, which, in turn, starts its engine. The engine then engages a clutch that connects the engine to the flywheel and generator, and continues to operate the unit until the utility returns to normal operating condition.

NetApp
Traditionally designed cooling systems indeed are the most significant energy wasters in a data center, indicates Ralph Renne, director of site operations for data storage product and service provider NetApp Inc., Sunnyvale, CA. The company is designing a 14,000–square foot engineering data center for energy efficiency using features such as environmentally friendly flywheel UPS systems, energy-efficient transformers, outside air economizers, and a variable primary chiller plant. The local power company, Pacific Gas and Electric Company (PG&E) presented NetApp with a rebate of about $1.43 million under PG&E’s Non-Residential New Construction Program, the largest new construction incentive that it has ever awarded. The company will save an estimated 11.1 million kWh each year, for a savings of more than $1.1 million, and a reduction of carbon dioxide emissions by 3,391 tons annually. The facility is expected to operate at a PUE of less than 1.3.

“One of the areas in which we achieved a significant portion of this rebate was fundamentally in the mechanical design; I’d say that 90% of the rebate dollars were associated with the energy efficiency measures we implemented from the mechanical cooling,” says Renne, adding that the facility infrastructure was ready in August 2008, and full migration of IT equipment would be completed two or three years later.

A key component of the energy-efficient system was the use of outside air handlers equipped with economizers that bring in outside air for cooling when outside temperatures are favorable. “One of the fundamental ways in which we can achieve efficient mechanical cooling is to not have a mechanical cooling system but, rather, bring in outside air when temperatures allow it and rely on the concept of free cooling,” he says. “Any time the temperature outside is less than 70 degrees Fahrenheit, we can be in a full mechanical economizer mode and effectively shut off our mechanical cooling. Bringing in the outside air allows us to operate very efficiently for greater than 6,000 hours a year—roughly three-quarters of the year.”

To address concerns about humidity and particulates, Renne says, the facility is equipped with a double-filtration system consisting of an exterior louver wall and an economizer damper. The outside air is brought into the facility, where it is delivered to the equipment before an exhaust fan exhausts the heat from the roof of the facility. This system necessitated the construction of a centralized shaft in the building, Renne notes. “We have a convection effect where heat is naturally rising—this is simply pulling some of that heat out and exhausting it, fan-assisted,” he says.

Photo: Advanced Data Centers View of outside air louvers used to cool the interior of Advanced Data Centers’ first data center

The new facility will have 720 racks of IT equipment and consume about 5.76 MW of power. Cooling it will require not only the creative use of outside air, but also a cold-aisle containment system in which the fronts and back of the racks face each other, says Renne. “Containing the hot and cold aisles allows us to eliminate mixed air,” he adds.

Cold aisles will be contained and the air pressurized, using VFD-equipped exhaust fans. Renne explains that this process will allow the minimization of fan energy, as well as the monitoring of hot- and cold-aisle differential pressure at exhaust ducts.

“We’re able to effectively achieve a 25-degree delta-t [temperature differential] across the coil, meaning the coil is working to its design capacity,” says Renne. “The industry used to believe that IT equipment needed 52-degree air. This is a paradigm shift that I would say has taken place over the past eight to 10 years; IT equipment can handle a much warmer environment. Data center operators fundamentally believed a [myth] that the colder, the better. In the winter, we’re generally below 70 degrees, and it simply allows us to operate our economizers in a much broader range and better leverage the free-cooling concept.”

Photo: Sun Microsystems Power distribution boxes can be twisted into an overhead power system, reducing the amount of copper wire needed by about half at Sun’s Broomfield facility.

Renne reports that NetApp adopted a philosophy favoring overhead HVAC for data centers long ago. “A raised-floor distribution system has two compounding problems: hot air rises and cold air falls, so a natural benefit of physics is to remove the hot air from overhead and distribute the cold air overhead which allows us to have the convection working in our favor,” he says. “Trying to deliver cold air through a floor and also pull the hot air through the floor has its inherent problems. Raised-floor environments are something we moved away from many years ago.”

Geography plays a role in the move away from raised floors. Renne points out that a 4-foot-high raised floor that was required for NetApp’s nearby enterprise data center was very expensive to construct, given its location in seismic Zone 4.

Another key design feature of the Sunnydale facility is the use of a vari-prime chilled-water distribution system in conjunction with four 600-ton Trane centrifugal chillers, one of which is used as a backup in a system that requires 1,800 tons of chilled water. Renne says that the system allows chiller water temperature modulation at variable flow rates.

“What it allows you to do is reduce the amount of flow under low-load conditions,” he explains. “That’s an advantage, because our pumps can scale back and don’t have to flow at a constant rate. In an efficiency equation, a traditional primary-secondary chiller operates within the 0.53- to 0.56-kW-per-ton range; our chillers are operating in the 0.49-kW-per-ton range.”

Sunnyvale is another data center that relies upon a flywheel-based UPS system consisting of two 900-kV-Amps (kVA) flywheels. The facility is equipped with this type of system “for several reasons,” says Renne. “One, there’s a big efficiency gain with flywheels.”

The flywheels in the facility are 97–98% efficient, which represents an increase of about 6–8% over a traditional wet cell battery inverter system, according to Renne. “But there’s another benefit in the sense that, to have 1,800 kVA of backup capacity in the form of batteries would require a relatively large room, perhaps a dedicated space of 1,200 square feet with associated exhaust fans to remove the off-gassing of the batteries, and that real estate costs an enormous amount of money, especially out here in Sunnyvale. The ancillary benefit is that you don’t have a hazardous material and the costs and concerns with maintaining a battery room, which means you’ve got to replace the batteries within a three- to five-year window.”

A major difference between the Sunnyvale data center UPS and that of other centers, Renne says, is that his flywheels are decoupled from a 2,000-kVA generator serving the IT load and a 1,500-kVA generator serving the HVAC load, whereas many other data centers’ flywheels and generators are coupled. Sunnyvale’s flywheel bridges the time between a utility blackout to the full-speed operation of the generators. At their full design load, the flywheels have stored energy that can support the load for 13-and-a-half seconds. Normally, it takes seven-and-a-half seconds to get the diesel generators up to full power and there is a two-second time delay to allow for power sags.

The facility also uses Powersmiths K4-rated transformers, which, according to Renne, are about 1.5% more efficient than NEMA TP-1-rated transformers. Based on readings on the facility equipment when it resided in another location, he says, Total Harmonic Distortion (THD) was about 10% and K-rated transformers were required in the event that THD exceeded 5% anyway. “The premiums we paid to purchase these transformers really had a compelling return on investment in that they would pay for themselves in less than a one-year timeframe based on full load once the IT space is fully occupied,” says Renne. “It wasn’t too hard to sell the idea to the project management and finance guys, because it had an inherent payback. Then, you’ve got the ongoing savings, and, honestly, the life cycle is probably 20 years for these components, so if we did a life-cycle cost analysis, it would be an enormous amount of savings.”

DG Applicability
The managers of these projects say that a data center is just one example of a facility that is well-suited to a DG system, which can take a burden off of the local power grid. Monroe indicates that using a DG system would be a good fit for data centers, given the move toward energy efficiency. Citing examples such as natural gas-powered microturbines, natural gas–powered fuel cells, solar, wind, or even methane gas from landfills, he argues that DG energy production can be more efficient than the grid.

“A great thing about data centers is that the loads are typically high and steady,” points out Monroe. “The variation in loads is not very great, so a fuel cell or a microturbine power generation system fit hand-in-glove with data centers. Folks are even looking at a [CHP] system as the primary source, and the grid as the backup, instead of the way people do it today—where the grid is primary and we have diesel generators or other systems as the backup.” Monroe adds that he often cites a 2007 EPA CHP cost-savings comparison (see Table 1) in presentations, in which he points out that only about 15% of grid power is available to a data center after transmission, compared with that produced by an onsite gas turbine that might be twice as efficient.

Seese agrees that CHP, in particular, makes sense for onsite data center power production. “In the end, we’re all trying to accomplish the same thing—we’re all trying to get the same benefit from each watt of power that we use,” he says. “Certainly, the CHP approach, where you’re using the heat to cool the facility to run chillers, is a great solution and one that makes a lot of sense when you look at the ability to cool onsite. A lot of the concepts we’re using are the same: reusing the heat in an effective manner. The CHP approach uses the exhaust heat from the electrical plant to chill the facility. I’m very concerned that the generating plant is generating 1 watt of power, and by the time it gets to me it’s one-third of a watt. We’re using two-thirds of our power just in that distance it has to travel in the various substations. I would love to reduce those losses between generation and load. We’re all hopeful that the idea of cogeneration or onsite generation will become more affordable, because it only makes sense in these data centers. We really need to focus more and more on energy savings and onsite generation is going to be part of our solution, I’m convinced.”

NetApp has experience with DG for data centers. Its enterprise data center has a CHP system that uses a 1,125-kW Hess Microgen cogeneration plant. Waste heat energy is used to produce up to 300 tons of chilled water via adsorption chillers. Renne notes that the system does not serve as a primary power source, but reduces the amount of power necessary from the local utility. During the summer rate season, he says, the company can operate the cogen plant cost effectively.

“It may be a small percentage of total load, but some companies are starting to look at where they can incorporate renewable energy or distributed generation into their data centers,” says Renne. “We’ve done some internal consideration of fuel cells. We look at a fuel cell, and its relatively high density, and—in a compact footprint—it could certainly be utilized to supplement some import of utility power. With the fact that you have a base load, you can really start to look at distributed generation as a whole and have a relatively reliable payback model that I’d say traditional office space doesn’t offer, in the sense that there’s a load factor that’s varying by a very large percentage.”