IT facilities, which are making great strides in improving energy efficiency, are a natural fit for onsite power generation.
Address one environmental
challenge and sometimes another materializes. The e-communications revolution
that has occurred over the past decade-plus has saved innumerable trees and
reduced burdens on landfills and recycling facilities, in addition to
exponentially improving the efficiency of information flow. But as technology
applications such as Web hosting, financial transactions, and medical record
storage continue to increase the need for electronic processing, storage, and
networking equipment, energy demand to power it has increased exponentially.
The EPA estimates that data
centers consumed 1.5% of the nation’s power, or 61 billion kWh, in 2006, and
these facilities’ share will be expected to have roughly tripled by the end of
the decade. EPA also estimated that the amount of power consumed by these
facilities in 2006 was double the amount in 2000, and that if current efficiency
trends continued, total data center power consumption would reach about 100
billion kWh by 2011. The US Congress has taken note: Public Law 109-431 required
EPA to conduct a study of the rapid growth of energy consumption in data centers
and recommend actions to reduce their energy consumption. An August 2007 EPA
report to Congress provided several recommendations for improved operation, best
practices, and the state of the art. Some of the recommendations relating to
power and cooling include improvements in fans, chillers, and pumps, and the use
of free cooling and combined heat and power (CHP).
Several large technology
corporations have followed suit and made energy efficiency improvements in their
data centers, including Hewlett-Packard, IBM, Google, and Sun Microsystems. Just
as significantly, the information technology (IT) industry has developed its own
metric for data center power consumption. The Green Grid, an association of IT
professionals focusing on the energy efficiency of data centers, proposed the
use of Power Usage Effectiveness (PUE), or total power consumption divided by IT
equipment power consumption, and the metric has been widely adopted by the IT
industry.
Recently, three companies have
successfully reduced their data center power consumption using this metric and
EPA and industry recommendations. These projects bode well for the future of
distributed generation (DG) in general.
 |
Photo: NetApp
Interior view of a louver wall at the new NetApp facility, which
is used to bring outside air into the facility for cooling. |
Sun Microsystems
A recent move by Sun Microsystems
to combine two data centers into one in Broomfield, CO, will save the company 1
million kWh of electricity per month and an estimated $1 million in energy costs
annually. Design strategies, in the largest data center consolidation project
undertaken in the company’s history, included minimizing the raised floor space
and optimizing chilled-air distribution. The Santa Clara, CA–based company
opened the facility in January 2009 and reduced total floor space by about 75%,
eliminated $4 million in costs with the raised-floor reduction, saved more than
$1 million in electrical costs, reduced its carbon dioxide (CO2)
emissions by 11,000 metric tons per year, and is expected to cut its corporate
carbon footprint by 6%.
The facility is the company’s
fourth to use a new energy-efficient design, following in the footsteps of data
centers in Santa Clara; Blackwater, UK; and Bangalore, India. The Santa Clara
facility, which opened in August 2007, was compressed from 200,000 to 80,000
square feet—at a $1-million savings and yielding a PUE of 1.28. Mark Monroe,
director of sustainable computing for Sun Microsystems, acknowledges that the
company saw an opportunity for cost savings in the wake of the other facilities’
construction and the 2005 acquisition of server storage device provider Storage
Technology (StorageTek).
Monroe reports that Sun had spent
about $250 million over the previous four years for an initiative to reduce the
company’s data center floor space by about one-third. The primary motivation for
modifying the acquired StorageTek facility in nearby Louisville, CO, was cost
savings, with sustainability emerging in conjunction, Monroe says. “I like to
say that we’re pragmatic ecologists, and I think in every for-profit company
you’ll find will say the same thing,” he says. “Here in Colorado, we get our
power from two coal-fired power plants just down the street. That 11,000
metric-tons [CO2 reduction] is about 5% of our global carbon
footprint, and about 6% of our US carbon footprint.”
The Broomfield consolidation is
part of a larger Sun initiative to achieve a 60% data center square footage
compression globally. This initiative has included a 30% decrease in data center
operating expenses in the Bay Area alone since 2007. In October 2008, Sun
announced that it had reduced greenhouse gas emissions from US operations by
23%, surpassing its goal five years early; the company is now attempting to
reduce its emissions by another 20%, with projects like Broomfield. Using its
experience in improving energy in its own data centers, Sun recently launched
its own design service for these facilities.
The equipment in an acquired
496,000–square foot lab and data center in Louisville was moved to Sun’s campus
in nearby Broomfield, where overall IT equipment floor space was compressed to
126,000 square feet and 165,000 square feet of raised floor was condensed to 700
square feet, says Monroe. This reduction alone reduced much of the facility
costs, because it eliminated the need for additional reinforcement to support
the weight of the servers, which can weigh as much as 1 ton per rack, according
to Monroe. Noting that raised floors have traditionally served dual purposes of
facilitating the distribution of both chilled air and cabling, he says,
“Unfortunately, those two purposes are at odds with each other because the more
cabling you put underneath the floor, the more it restricts the airflow, and the
more problems you have with the cooling system because the air becomes
unbalanced—you have hot spots.”
The facility’s cooling system uses
containment of hot air produced by the servers within the aisles. Another
feature of the design, notes Monroe, is the close proximity of the cooling
equipment to the IT equipment; in many cases, the hot air only travels 2 or 3
feet before being blown across cooling coils, and gets blown around to the front
of the computer. The facility utilizes “close-coupled cooling” with Liebert
Corp. overhead cooling systems and APC row-level cooling systems. The design and
equipment are much more efficient than a traditional data center design that
allows hot air to meander to a register all the way across the room before being
blown under a raised floor, Monroe points out.
Additionally, the facility is
equipped with two Trane variable-speed centrifugal 500-ton chillers with
two-stage compressors that are 25% more efficient than American Society of
Heating, Refrigerating, and Air-Conditioning Engineers standards. “We also use
quite a bit of evaporative cooling; rather than run the chillers here in
Colorado on a day when it’s 45°F and gets down to the twenties at night, we use
a flat plate evaporator out in the cooling tower to do the heat exchange, and we
can get over 1,000 hours of cooling each year,” says Monroe. The cooling system
uses a Clearwater Dolphin closed-loop electromagnetic purification system for
the chiller water that saves the facility about 675,000 gallons of water
annually, according to Monroe.
 |
Photo: NetApp
Powersmiths transformers are being used to increase energy
efficiency at a new NetApp Inc. engineering and data center. |
The cabling in the facility also
has been reconfigured according to Sun’s pod architecture, another design item
that reduced costs, Monroe adds. The networking and power distribution cables
are located above the server racks and power distribution equipment is now
located above the server racks in what is described as a pod design. Power
distribution boxes such as three-phase, 208-V or single-phase, 110-V can be
twisted into an overhead Universal Electric Corp. Starline Plug-In Busway
system, reducing the amount of copper wire needed by about half, according to
Monroe. “With a raised floor, no one ever pulls cable out,” he says. “It’s
either too long or too much work, so when modifications are made over the life
of a data center, copper cable builds up underneath there in the form of
networking cables and power cables. With this overhead system, every time we
make a modification, we take the old cable out, we put the new cable in, and
we’re going to save tons of copper inside the data center, which is a scarce
resource these days.”
Another energy saver is the use of
flywheel-based UPS rather than traditional lead acid batteries, Monroe points
out. “Number one, they run more efficiently all the time, and number two, they
don’t have this room full of lead acid batteries that need to be replaced every
six years, meaning that you’ve got to dispose of the lead and chemicals,” he
says. Monroe recalls a day in February 2009, when high winds caused a power
outage but the flywheels started up the backup generators, which provided
uninterrupted power for a couple of hours.
Advanced Data Centers
A multidisciplinary team
engineering approach to improving energy efficiency resulted in a PUE that is
pushing the limit of theoretical achievement for San Francisco, CA-based
Advanced Data Centers’ first data center on the former McClellan Air Force Base
near Sacramento, CA. The company will provide facility space and equipment for
customers that opt to not operate their own data centers. By exhausting
equipment-generated heat, making the distribution of chilled water more
efficient, and other methods, the team is expected to obtain a PUE of 1.12 when
the 200,000–square foot facility is completed in late 2009. The design is
estimated to provide about $2 million in energy savings every year, a fact not
lost on the Sacramento Municipal Utility District, which awarded the
three-year-old startup with its largest-ever rebate under the “Savings by
Design” energy efficiency incentive program.
Bob Seese, the company’s chief
data center architect, says that the multidisciplinary team approach to facility
design that came about in 2007 was a result of the movement toward energy
efficiency in data centers.
“At that time, the industry,
although it was starting to migrate in that direction, still thought that
electric costs and the electricity that was being used by data centers were
untouched areas because a data center had to remain operational 24 hours a day,
seven days a week, and 365 days a year, and touching energy could, they felt,
impact its reliability,” says Seese. “Understanding that that was the industry’s
position, I had to support my position. I thought that there was a lot of
opportunity for savings in terms of cooling.”
Seese put together a team of
experts in multiple engineering disciplines including electrical, mechanical,
and structural. Significantly, he adds, collaborations were based on the Rocky
Mountain Institute’s charette concept. “We instructed everyone when they came to
these charettes to leave their particular specialty at the door and, instead,
come into the room and be an engineer that day—not necessarily a structural,
civil, or mechanical engineer.”
He credits the approach as
instrumental in helping the team to get the facility pre-certified for a US
Green Building Council’s Leadership in Energy and Environmental Design Core and
Shell Platinum rating.
 |
Photo: NetApp
A vari-prime chilled-water distribution system is used in
conjunction with these Trane chillers to efficiently cool the NetApp facility. |
“They really did find creative
solutions instead of just going back to the old habits of doing things the way
they had always been done,” says Seese. “A lot of that was based on the naïveté
of engineers, who maybe didn’t understand mechanical engineering, but they
asked, ‘Why don’t you do it this way?’ And in a lot of cases, the mechanical or
electrical engineer would come back and say, ‘Let me see if that solution does
work.’”
One of the most important
strategies that the team used, Seese points out, was making use of the outside
air temperature in the Sacramento area. “We looked at the weather in the
Sacramento area and found out that we could use outside air to cool our facility
75% of the year,” he says. “In a typical data center, chillers consume as much
as 0.6 kW per ton—that’s a lot of energy. If we could find a way to use outside
air and not cause problems with the dust count in the air, we could save a lot
of energy.”
Seese adds that Rumsey Engineers
of Oakland, CA, which had assisted in the design of many “clean rooms” for
semiconductor manufacturing, had successfully utilized outside air for cooling
these facilities.
Average high temperatures in the
Sacramento area range from 55–82°F between November and May, compared with
89–94°F from June through September. The bigger challenge in bringing outside
air into the new facility via high-efficiency fans, Seese says, was controlling
the humidity for a few days out of the cooler months—without consuming a lot of
power. Noting that the region usually has optimal humidity for the natural
cooling strategy, he reports that high-efficiency atomizing dehumidifiers will
be used to dehumidify the return air stream.
During months in which it was
necessary to chill water for cooling, the facility will use up to six modular
centrifugal chillers: five 750-ton units and a 250-ton unit for low-load
conditions for initial occupation of the space. The 250-ton unit and one of the
750-ton units are equipped with variable frequency drives; Seese explains that
these two units would be used in situations where 850–1,500 tons of chilled
water are needed, and the other 750-ton units can be utilized incrementally when
more than 1,000 tons are needed. Another significant cost-saving aspect of the
facility, Seese adds, is the fact that the chillers are located outside of the
facility. “We had to put our chillers outdoors because putting our chillers
indoors would have had too much of an impact on the amount of real estate we had
to lease out for business purposes—we had a certain number of square feet to
rent,” he notes. “They don’t like the rain, so you have to cover them; this
solution puts the cooling towers over the top of the chillers.”
“This idea of delivering cold air
underneath the floor and expecting that cold air to rise runs counter to the
laws of physics,” says Seese of traditional data center cooling system design.
He explains that the team focused on minimizing the number of required turns in
the distribution of chilled air.
In contrast to a typical data
center—where air is delivered beneath a raised floor, makes immediate
right-angle turns, and travels across the floor before making another
right-angle turn, and finds its way up through perforated floor tiles in front
of the computer equipment—the new facility will draw air through the outside air
louvers, and the drawn-in air will follow a straight path through filters and
cooling coils to the equipment floor. The equipment floor is pressurized so that
the temperature of the air from the floor to the ceiling is constant—eliminating
the hot and cold air mixing that is found in typical data centers.
Additionally, the team designed a
“hot-aisle containment system” that pressurizes air and exhausts
equipment-generated heat from the facility to prevent it from mixing with the
cooler air. Doors were installed at the end of hot aisles to contain the heat
produced by the equipment and blown out the back of the equipment by fans. “Cold
aisles” are in the front of the equipment and back up to an enclosure that
captures and exhausts the hot air. Roofs were constructed above the servers and
chimneys were constructed on top of the roofs. Above the chimneys are plenums
with variable frequency drive (VFD)-equipped fans that draw the air in the hot
aisles upward and exhaust it.
“At certain times of the year, the
[outside] air is too cold, so we allow the hot air to come back in and mix with
the cold air coming in from outside to temper it down to 70 to 72 degrees,” says
Seese. “The other thing it does is allow us to cool air based on its
temperature. The outside air can get to 104 degrees, and our return air could be
95-degree air. It makes more sense for us to cool the 95-degree air, because it
requires less energy.
 |
Phot:o: NetApp
Exterior view of louver wall used to bring filtered air into the NetApp
facility for cooling |
“The other thing we’ve done is
completely eliminate 90-degree turns in water delivery, and that is obviously
done for the purpose of making the water run much more efficiently and smoothly,
and reduces our pump energy significantly,” adds Seese.
The resulting higher efficiency,
he says, allows the delivery of 55°F water to the chillers, rather than the
typical 45°F water. Energy consumption is 0.32–0.35 kW per ton, rather than the
typical 0.6 kW per ton. Overall, Seese notes, all of the power consumption,
except for 12%, is for the purpose of powering the IT equipment.
The facility also features a UPS
that relies on a 3-ton flywheel, rather than Direct Current (DC) batteries for
backup power. Seese points out that the flywheel is about 3% more efficient than
the best available batteries, which are about 94% efficient, according to a
Lawrence Berkeley National Laboratory survey relating to double-conversion UPS
systems. The flywheel is more efficient than batteries and allows conversion of
alternating current from the grid to DC, exclusively for the data center
equipment, Seese says. A Hitec motor/generator that is normally powered by a
motor at the local utility would be powered by the flywheel at the data center.
“The typical data center uses alternating current, turns it into direct current,
turns it back into alternating current, and sends it up to the servers, which
immediately turn it into direct current,” says Seese, adding that this occurs to
charge the batteries and correct any anomalies in the power that will be
delivered to the servers.
In addition to turning the
generator, the utility power also spins the 3-ton flywheel, which keeps spinning
via its own inertia and continues to spin the generator, which, in turn, starts
its engine. The engine then engages a clutch that connects the engine to the
flywheel and generator, and continues to operate the unit until the utility
returns to normal operating condition.
NetApp
Traditionally designed cooling
systems indeed are the most significant energy wasters in a data center,
indicates Ralph Renne, director of site operations for data storage product and
service provider NetApp Inc., Sunnyvale, CA. The company is designing a
14,000–square foot engineering data center for energy efficiency using features
such as environmentally friendly flywheel UPS systems, energy-efficient
transformers, outside air economizers, and a variable primary chiller plant. The
local power company, Pacific Gas and Electric Company (PG&E) presented
NetApp with a rebate of about $1.43 million under PG&E’s Non-Residential New
Construction Program, the largest new construction incentive that it has ever
awarded. The company will save an estimated 11.1 million kWh each year, for a
savings of more than $1.1 million, and a reduction of carbon dioxide emissions
by 3,391 tons annually. The facility is expected to operate at a PUE of less
than 1.3.
“One of the areas in which we
achieved a significant portion of this rebate was fundamentally in the
mechanical design; I’d say that 90% of the rebate dollars were associated with
the energy efficiency measures we implemented from the mechanical cooling,” says
Renne, adding that the facility infrastructure was ready in August 2008, and
full migration of IT equipment would be completed two or three years later.
A key component of the
energy-efficient system was the use of outside air handlers equipped with
economizers that bring in outside air for cooling when outside temperatures are
favorable. “One of the fundamental ways in which we can achieve efficient
mechanical cooling is to not have a mechanical cooling system but, rather, bring
in outside air when temperatures allow it and rely on the concept of free
cooling,” he says. “Any time the temperature outside is less than 70 degrees
Fahrenheit, we can be in a full mechanical economizer mode and effectively shut
off our mechanical cooling. Bringing in the outside air allows us to operate
very efficiently for greater than 6,000 hours a year—roughly three-quarters of
the year.”
To address concerns about humidity
and particulates, Renne says, the facility is equipped with a double-filtration
system consisting of an exterior louver wall and an economizer damper. The
outside air is brought into the facility, where it is delivered to the equipment
before an exhaust fan exhausts the heat from the roof of the facility. This
system necessitated the construction of a centralized shaft in the building,
Renne notes. “We have a convection effect where heat is naturally rising—this is
simply pulling some of that heat out and exhausting it, fan-assisted,” he
says.
 |
Photo: Advanced Data Centers
View of outside air louvers used to cool the interior of Advanced Data
Centers’ first data center |
The new facility will have 720
racks of IT equipment and consume about 5.76 MW of power. Cooling it will
require not only the creative use of outside air, but also a cold-aisle
containment system in which the fronts and back of the racks face each other,
says Renne. “Containing the hot and cold aisles allows us to eliminate mixed
air,” he adds.
Cold aisles will be contained and
the air pressurized, using VFD-equipped exhaust fans. Renne explains that this
process will allow the minimization of fan energy, as well as the monitoring of
hot- and cold-aisle differential pressure at exhaust ducts.
“We’re able to effectively achieve
a 25-degree delta-t [temperature differential] across the coil, meaning the coil
is working to its design capacity,” says Renne. “The industry used to believe
that IT equipment needed 52-degree air. This is a paradigm shift that I would
say has taken place over the past eight to 10 years; IT equipment can handle a
much warmer environment. Data center operators fundamentally believed a [myth]
that the colder, the better. In the winter, we’re generally below 70 degrees,
and it simply allows us to operate our economizers in a much broader range and
better leverage the free-cooling concept.”
 |
Photo: Sun Microsystems
Power distribution boxes can be twisted into an overhead power
system, reducing the amount of copper wire needed by about half at Sun’s Broomfield facility. |
Renne reports that NetApp adopted
a philosophy favoring overhead HVAC for data centers long ago. “A raised-floor
distribution system has two compounding problems: hot air rises and cold air
falls, so a natural benefit of physics is to remove the hot air from overhead
and distribute the cold air overhead which allows us to have the convection
working in our favor,” he says. “Trying to deliver cold air through a floor and
also pull the hot air through the floor has its inherent problems. Raised-floor
environments are something we moved away from many years ago.”
Geography plays a role in the move
away from raised floors. Renne points out that a 4-foot-high raised floor that
was required for NetApp’s nearby enterprise data center was very expensive to
construct, given its location in seismic Zone 4.
Another key design feature of the
Sunnydale facility is the use of a vari-prime chilled-water distribution system
in conjunction with four 600-ton Trane centrifugal chillers, one of which is
used as a backup in a system that requires 1,800 tons of chilled water. Renne
says that the system allows chiller water temperature modulation at variable
flow rates.
“What it allows you to do is
reduce the amount of flow under low-load conditions,” he explains. “That’s an
advantage, because our pumps can scale back and don’t have to flow at a constant
rate. In an efficiency equation, a traditional primary-secondary chiller
operates within the 0.53- to 0.56-kW-per-ton range; our chillers are operating
in the 0.49-kW-per-ton range.”
Sunnyvale is another data center
that relies upon a flywheel-based UPS system consisting of two 900-kV-Amps (kVA)
flywheels. The facility is equipped with this type of system “for several
reasons,” says Renne. “One, there’s a big efficiency gain with flywheels.”
The flywheels in the facility are
97–98% efficient, which represents an increase of about 6–8% over a traditional
wet cell battery inverter system, according to Renne. “But there’s another
benefit in the sense that, to have 1,800 kVA of backup capacity in the form of
batteries would require a relatively large room, perhaps a dedicated space of
1,200 square feet with associated exhaust fans to remove the off-gassing of the
batteries, and that real estate costs an enormous amount of money, especially
out here in Sunnyvale. The ancillary benefit is that you don’t have a hazardous
material and the costs and concerns with maintaining a battery room, which means
you’ve got to replace the batteries within a three- to five-year window.”
A major difference between the
Sunnyvale data center UPS and that of other centers, Renne says, is that his
flywheels are decoupled from a 2,000-kVA generator serving the IT load and a
1,500-kVA generator serving the HVAC load, whereas many other data centers’
flywheels and generators are coupled. Sunnyvale’s flywheel bridges the time
between a utility blackout to the full-speed operation of the generators. At
their full design load, the flywheels have stored energy that can support the
load for 13-and-a-half seconds. Normally, it takes seven-and-a-half seconds to
get the diesel generators up to full power and there is a two-second time delay
to allow for power sags.
The facility also uses Powersmiths
K4-rated transformers, which, according to Renne, are about 1.5% more efficient
than NEMA TP-1-rated transformers. Based on readings on the facility equipment
when it resided in another location, he says, Total Harmonic Distortion (THD)
was about 10% and K-rated transformers were required in the event that THD
exceeded 5% anyway. “The premiums we paid to purchase these transformers really
had a compelling return on investment in that they would pay for themselves in
less than a one-year timeframe based on full load once the IT space is fully
occupied,” says Renne. “It wasn’t too hard to sell the idea to the project
management and finance guys, because it had an inherent payback. Then, you’ve
got the ongoing savings, and, honestly, the life cycle is probably 20 years for
these components, so if we did a life-cycle cost analysis, it would be an
enormous amount of savings.”
DG Applicability
The managers of these projects say
that a data center is just one example of a facility that is well-suited to a DG
system, which can take a burden off of the local power grid. Monroe indicates
that using a DG system would be a good fit for data centers, given the move
toward energy efficiency. Citing examples such as natural gas-powered
microturbines, natural gas–powered fuel cells, solar, wind, or even methane gas
from landfills, he argues that DG energy production can be more efficient than
the grid.
“A great thing about data centers
is that the loads are typically high and steady,” points out Monroe. “The
variation in loads is not very great, so a fuel cell or a microturbine power
generation system fit hand-in-glove with data centers. Folks are even looking at
a [CHP] system as the primary source, and the grid as the backup, instead of the
way people do it today—where the grid is primary and we have diesel generators
or other systems as the backup.” Monroe adds that he often cites a 2007 EPA CHP
cost-savings comparison (see Table 1) in presentations, in which he points out
that only about 15% of grid power is available to a data center after
transmission, compared with that produced by an onsite gas turbine that might be
twice as efficient.
Seese agrees that CHP, in
particular, makes sense for onsite data center power production. “In the end,
we’re all trying to accomplish the same thing—we’re all trying to get the same
benefit from each watt of power that we use,” he says. “Certainly, the CHP
approach, where you’re using the heat to cool the facility to run chillers, is a
great solution and one that makes a lot of sense when you look at the ability to
cool onsite. A lot of the concepts we’re using are the same: reusing the heat in
an effective manner. The CHP approach uses the exhaust heat from the electrical
plant to chill the facility. I’m very concerned that the generating plant is
generating 1 watt of power, and by the time it gets to me it’s one-third of a
watt. We’re using two-thirds of our power just in that distance it has to travel
in the various substations. I would love to reduce those losses between
generation and load. We’re all hopeful that the idea of cogeneration or onsite
generation will become more affordable, because it only makes sense in these
data centers. We really need to focus more and more on energy savings and onsite
generation is going to be part of our solution, I’m convinced.”
NetApp has experience with DG for
data centers. Its enterprise data center has a CHP system that uses a 1,125-kW
Hess Microgen cogeneration plant. Waste heat energy is used to produce up to 300
tons of chilled water via adsorption chillers. Renne notes that the system does
not serve as a primary power source, but reduces the amount of power necessary
from the local utility. During the summer rate season, he says, the company can
operate the cogen plant cost effectively.
“It
may be a small percentage of total load, but some companies are starting to look
at where they can incorporate renewable energy or distributed generation into
their data centers,” says Renne. “We’ve done some internal consideration of fuel
cells. We look at a fuel cell, and its relatively high density, and—in a compact
footprint—it could certainly be utilized to supplement some import of utility
power. With the fact that you have a base load, you can really start to look at
distributed generation as a whole and have a relatively reliable payback model
that I’d say traditional office space doesn’t offer, in the sense that there’s a
load factor that’s varying by a very large percentage.”