July-August 2010

Risk and Interruption

Onsite power plays a critical role in disaster recovery.

Tweet

News reports make it feel like the risk of business interruption from power outages due to storms or other types of natural or manmade disasters is on the rise. Whether or not they are truly becoming more frequent, there’s little doubt that dealing with these catastrophic events is becoming more and more costly for businesses and communities. In 2004, the International Red Cross published its annual World Disasters Report, in which it claimed that the economic cost of natural disasters had skyrocketed. Since 1980, direct economic losses from natural disasters multiplied fivefold to $629 billion. Annual direct losses from weather-related events increased from an estimated $3.9 billion in the 1950s to $63 billion in the 1990s.

Individual figures are equally alarming. A 2003 blackout in the Northeast affected millions of customers in eight states, with a cost estimated at $6 billion. Losses from Hurricane Andrew may exceed $25 billion. Losses from Hurricane Katrina, the 9/11 bombing, and earthquakes in Haiti and Chile are almost incalculable. Equally disturbing is the fact that, according to a US Geological Survey, insurance coverage for losses resulting from natural disasters is typically less than 20% of the total loss. The remainder of costs is covered by the federal government through emergency allocations, which ultimately increases the national debt.

The situation has reached such a serious level that in February, the Renewable Energy World Conference & Expo North America met to discuss disaster planning and the role that renewable energy and other energy sources can play in both short- and long-term recovery. As if to demonstrate the vulnerability of infrastructure in the face of disaster, on February 27, just days after the meeting, Chile experienced an earthquake of 8.8 magnitude.

Because the costs and risks of power outages have been made more urgent, companies are implementing disaster recovery plans that often feature onsite power generation. An added bonus of the plan is that some utility companies offer revenue opportunities. Another plus is that, often, the power produced onsite is cleaner.

So Easy, a Data Cave Can Do It
Columbus, IN, experienced a major 500-year flood in June 2008 that destroyed buildings and wreaked havoc on lives. “That kind of event makes people think about disaster recovery measures,” says Caleb Tennis, president of Data Cave, a full-service, Tier IV-compliant data center located in the city. “Sometimes you have to go through a disaster before you think about a disaster recovery plan—or, in some industries, there are regulations that force you to think about it. The health care industry mandates a disaster recovery plan and audits it.” He claims 80% of companies that experience a major loss go out of business.

Photo: CoSentry Based in Omaha, NE, CoSentry offers disaster recovery, hosting, co-location and managed technical services with an “always on, always connected” business model.

Data Cave is a design facility for housing critical IT equipment for companies that don’t want to spend money on infrastructure or choose to outsource management. Typical customers include municipalities, hospitals, IT solution providers, banks, and Fortune 500 companies. “You need two facilities in case one goes down,” points out Tennis. “The business model is old, but it’s capital-intensive to build: You need two generators and two of everything else, such as chillers, a/c [air-conditioners], and power distribution systems. You can’t have one fail to start.”

Most existing facilities are 10–15 years old, and don’t have the capabilities for new technology, says Tennis. “Computers run hotter than they did 10 years ago; you have to upgrade the building or move.”

He adds that the market is going to large-scale amounts of power and heat. “If you run 20 computers, you need generators, circuit breakers, UPS.… That’s expensive, but for $1,000 per month, you can outsource.”

Because the cost to build is high—Data Cave’s facility cost $100 million—not a lot of companies are building any more, so they outsource. Tennis says his customers “like the Midwest because it’s cheaper for space and power.”

The Data Cave facility was designed for 10 MW of incoming redundant utility power in a tornado-proof building. Each quadrant of the facility is serviced by completely independent equipment, including: two 2.5-millivolt-ampere electrical transformers, from separate utility feeds; two 2-MW Cummins diesel-powered generators; and electrical switchgear to control and balance the power, including transient surge suppression. In addition, each quadrant’s critical Information Technology (IT) load is backed up by two Active Power flywheel UPS systems, providing isolation from the incoming power grid and very clean sinusoidal output, protecting against spikes and surges in the incoming power.

“We provide 100% redundancy most of the time,” says Tennis. Providing disaster recovery options and strategies anywhere in the US, Data Cave offers hot site backup with real-time application for uninterrupted service and cold site backup with spare critical systems on hand in case of emergency.

Data Cave already had a 600-kilovolt-ampere (kVA) clean source UPS system, installed in 2008–09, but, due to growth, they needed additional capacity. Tennis did due diligence on vendors, but chose to stay with Active Power, upgrading to two 1,200-kVA units for over 1 MG of backup power and the modular capability to add on in the future. In fact, Tom Nester, Active Power regional sales manager, North Central territory, says he foresees three more similar build-outs in Data Cave’s future. Tennis confirms: “This is the first phase of our build-out; we will expand.”

Familiarity with the system wasn’t the only reason Tennis opted to stay with Active Power. Tennis, an engineer, evaluated systems and how they’re built. “Large battery systems have a disadvantage: They take up a lot of space, and there are environmental issues because they have to be replaced every few years. The reliability isn’t as high as we want, and if it’s an issue, you don’t notice it until you need it … when it’s too late.”

Batteries act as a “security blanket,” declares Nester, but he adds that “it’s not real security if something goes wrong.” He claims they provide a false sense of security and points to maintenance and power abnormalities. “Battery maintenance is an afterthought,” he says. Their power degrades over time; if unchecked, they may not have enough reserve to be effective.

“‘Dirty’ power makes a system want to go to battery power,” he continues. “Constant small hits drain batteries.”

But a flywheel doesn’t degrade over time. “It takes small hits and still provides power,” he says.

As pointed out by Lee Higgins, Public Relations manager at Active Power, the power grid is not getting any bigger, but the need for power is growing … and the quality of power is decreasing. “We provide quality power for our customers,” he says.

Knowing that Active Power does only flywheel systems, which is newer technology for critical power, Tennis says they’re ahead of the competition, “many of whom just replace batteries with flywheels.” He realizes it’s more efficient if it’s designed as a flywheel system: in fact, 97% efficient (versus 92% with a battery system). “That can be as much as $10,000- to $20,000-a-month difference in electrical power costs.” The flywheel is more expensive upfront, he acknowledges, but the payback is quicker.

The downside of the flywheel versus the battery, Tennis says, is that they don’t provide power as long. “A battery runs for five to 15 minutes; a flywheel has less stored energy and runs for only 15 to 60 seconds.”

Future plans include increasing run time from the current 15–25 seconds with a bigger flywheel, Higgins mentions. However, Nester explains, “Any good generator and plan should kick in within 10 seconds.”

In fact, he says, the largest industries relying heavily on generator power—the hospital and pharmaceutical industries—are mandated and audited: Power must kick on in 10 seconds. Therefore, “how much time do I really have?” becomes “how much time do I really need?”

Tennis understands the situation. “You have to design a system so the generators are up and running quickly. A lot of people want the insurance of the battery, but it’s not necessary. The flywheel bridges the gap of power delay until the backup is up.”

A feature Tennis likes—and wasn’t aware of until after installation—is the double conversion. The power coming in is 48-bolt three-phase, he says. The flywheel converts it to Direct Current (DC) power, and then converts it back to Alternating Current (AC) power on the output. “This is for efficiency and storage, but the added benefit is that it doesn’t directly connect to the equipment, so if a disaster occurs, the IT equipment doesn’t blow,” he says. “Little power spikes are filtered out, so it’s outputting a pure energy wave to the IT equipment.”

It’s also “virtually maintenance-free,” says Tennis. An annual oil change and new bearings every three years are the only regular maintenance required. “It takes only a couple hours to replace bearings, and it can be done without shutting down, due to the dual redundancy.”

The recent upgrade gave Data Cave two complete systems, with four flywheels. As Tennis says, “We have an extra flywheel so one can be shut down for maintenance.”

Nester points out that, during routine maintenance, “With battery systems, you have to take down the whole line. You can’t isolate; you must put it on bypass. Flywheel maintenance is very simple.”

Installation was also easy. “We did it ourselves, which is unusual, but we’re self-sufficient,” states Tennis.

Nester considers it a testament to the design. “Our system ships complete,” he says. “With battery systems, you have to synchronize start-ups. You could blow the transformer if it’s not done properly. This is plug-and-play, and easily acclimated to the customer’s site.”

COD: Container on Delivery
When Hurricane Katrina hit New Orleans, the Ochsner health facility was kept running by generators. However—according to www.facilitiesnet.com—Grant Walker, Ochsner’s system vice president of supply chain and support services, considered the immobility of the emergency power a problem. “Almost everything we had was fixed before [Katrina],” he says. “We couldn’t move it anywhere; it was stuck.”

Since the disastrous hurricane, the facility’s generators come equipped with flatbed trucks, allowing the flexibility to move the emergency power within the seven-hospital health system. While the main function of the generators had been to power the hospital’s life-safety equipment before Katrina, the goal since then has been mobility. The organization has added mobile generators so most of the main campus has 100% emergency power.

Mobility is a critical feature during emergencies. Nester says Active Power’s flywheel systems are the answer to disaster planning because they’re quickly deployable. The complete turnkey solution includes a flywheel, switchgear, and generator in a pre-engineered, pre-wired, prefabricated container that can be deployed in weeks (versus months). “The flywheel is a perfect fit for container systems,” he says. “It has a small footprint and is rugged. It can handle any temperature while providing the same reliability.”

“They could be deployed in Haiti or Chile,” believes Higgins.

Anticipating increased acceptance of container systems in the US, he says these stackable units are suitable for more than just disaster recovery. With a custom paint job or company logo that doesn’t take away from the aesthetics, they can blend into the scenery when used to support utilities at events or to supplement infrastructure. For the time being, however, they are “mostly associated with data centers,” where they are providing green, reliable, efficient power more economically.

Resiliency Versus Redundancy

Some experts are shifting the focus of emergency contingency planning to resiliency over redundancy, because they believe the goal is to ensure that a facility can “weather a storm” or survive a crisis. In many cases, resiliency translates to duplication of equipment and processes, but it could involve alternative methods as well, depending on cost and risk tolerance.

Because data centers have a low tolerance for outages, most invest in costly redundant backup systems that include onsite power generation. Some companies are waking up to the fact that prevention is a critical part of disaster planning. “It’s becoming more important to avoid, prevent, or mitigate risks than [focus on] recovery,” states Kevin Dohrmann, chief technical officer with CoSentry.

Based in Omaha, NE, CoSentry offers disaster recovery, hosting, co-location and managed technical services with an “always on, always connected” business model. Downtime equals devastation for their clients, who include medical, insurance, banking/financial, law enforcement, transportation, and energy companies. For them, says Brian Driscoll, branch site manager, “redundancy is huge. A data center has no tolerance for unscheduled downtime; there’s almost no tolerance for scheduled downtime.” CoSentry requires “100% uptime. That’s what the customer wants. When you go down, they leave or get mad. If you’re down, you’re out of business.”

Fortunately, they’ve never lost power. However, plans to expand into unused space within its Sioux Falls facility increased their requirement for UPS power. Their existing Eaton Powerware Plus UPS, with 150-kVA capacity, couldn’t keep pace with CoSentry’s growth. “We outgrew the capacity of the unit; it was time to expand,” summarizes Dohrmann.

Driscoll adds that they worked on process for 14 months. “We weren’t on fire, per se, taking the time to be thorough.”

However, one week after installing their new system, EROS, US Geological Survey group lost its only UPS, endangering heavily used systems that provide access to 35 years of satellite images and 70 years of photos. “It confirmed our fears: We can’t let the clock tick,” says Driscoll. Their needs were for immediate added capacity, as well as room to grow and more redundancy.

Not only did their need for power per square foot grow exponentially, but CoSentry had outgrown its old system in other ways, says Larry Serbousek, front line sales with Innovative Electrical Technology America, an Eaton partner located in Omaha. “It was 12 years old,” he says. “Traditional electrical equipment lasts a long time, but in the current environment, with embedded electronics, life is shorter: 20 years old is obsolete.”

While their legacy system wasn’t exactly obsolete, it wasn’t as
efficient as current models. Serbousek estimates that old units were 80% efficient, while new units are 94–95% efficient, resulting in significant energy savings. Driscoll says CoSentry’s goal was to gain more power protection and significantly increased efficiency with less capital outlay.

The choice was Eaton’s 9395 UPS. Driscoll estimates that their old UPS was 85–90% efficient, due to “age and old technology,” and says the new system ranks in the mid-90 percentile for efficiency. By operating at greater than 94% efficiency, the 9395 reduces utility costs, creates cooler operating conditions, enhances reliability, and extends the overall life of the UPS components.

There’s potential to increase efficiency by using the Eaton Energy Saver System (ESS), which Serbousek says allows the 9395 to reach 99% efficient efficiency at all load levels. Using ESS, the UPS adapts to utility power conditions while supplying clean power, although it doesn’t offer precision of power, Serbousek says. “It regulates output to +/- 4% in regular mode and it follows the utility +/- 6-13% in energy saver mode, but you give up precise power.”

Driscoll hasn’t used the ESS yet. Nevertheless, the 9395 provides about a 10% gain in efficiency over CoSentry’s previous unit, with the potential for a 15% increase when he does use ESS mode. The energy saved during ESS usage typically allows customers to recover 100% of the UPS cost over a three- to five-year time period. At 250 kW of critical load, the savings represent about $4,000 per year per point of efficiency gain. “The payback numbers are unbelievable,” reiterates Serbousek.

Impressive as the savings are, the reliability is equally notable. The 9395 can be configured so its uninterruptible power modules automatically act as N+1 redundant systems, further enhancing reliability. “Older designs were non-redundant,” he says. “The new ones are internally redundant. That was a key issue that attracted them to Eaton.”

Internal redundancy was a factor in the decision matrix, along with cost, reliability, and modularity for future growth. Driscoll says a history with Eaton was no guarantee this time around; he was after affordable modularity. As manager of CoSentry’s smallest facility, he says he often inherits hand-me-down assets and has to make sure a dollar goes as far as possible. Having inherited an older system when CoSentry moved into the building and unaccustomed to having the opportunity to get new equipment, he wanted to make the right choice. “Old UPS units are traditional: non-modular. New ones are modular, and, soon, everyone will be making and buying modular. It’s basically two for the price of one, and it’s the same price kilowatt per kilowatt.”

CoSentry’s 500-kVA unit is configured with two 250-kVA UPMs that Driscoll says provide an N+1 redundancy on an 82-kW draw. Dohrmann says their selection provides the most capacity now and will allow them to increase the capacity by 50% in 2–3 years with the addition of a third unit, for which Driscoll says they have pre-planned space. “We expect to add a third module in a year or two,” says Driscoll. “Business is increasing. The economy affects us, but less than other industries. In good times, small- and medium-sized businesses are more easily able to take advantage of what we can provide. In tough times, large- and medium-sized businesses realize the benefits provided by our scale, experience, and resiliency-by-design.”

Driscoll also likes the modularity for serviceability. The redundant modules enable Eaton field technicians to completely isolate and service one module, while the others carry the load without going to bypass. Shifting load for maintenance reduces the risk for electricians—a consideration that’s important to him. “Around 90 people died in 2009 fixing UPS systems.”

He reckons that UPS is necessary 10 times a year, due to grid outages, brownouts, blips, surges, or other problems. “If the power company makes a mistake, if the power grid blows up, if there’s a delay in generator start-up, an accident, brownout, or power fluctuation, it isolates the data center from the ugliness upstream. But if there’s a failure in a non-modular system and that is all I have, I’m done.”

As Serbousek explains, a data center is flat-loaded all the time. With the modular design and internal redundancy, if one power module fails and the load is below half, it stays online. “Traditional UPS didn’t have that feature.”

Although modular design and room to grow were appealing attributes, other features could be considered essential. The unit gives CoSentry the built-in ability to do their own load bank testing. EZ Capacity Load Testing allows the unit to test its entire power train under load stress without requiring an external load bank.

Another beneficial feature is the eNotify Remote Monitoring Service, providing real-time monitoring of more than 100 UPS and battery alarms. “It sends me e-mails,” says Driscoll. He then can use his browser to go into the system to set thresholds, control, and see load; and check loss, status, and alerts. Both Driscoll and Dohrmann say the ease of monitoring and management has been one of the biggest changes, providing peace of mind.

Finding an environmentally friendly UPS that also saves money was critically important to Driscoll, who says the biggest job in retrofitting an existing site is building redundancy and resiliency. “Our motto is never build to set and forget. Any time there’s a new technology or advancement, we want to refresh our technology and own the advantage.”

Foreseeing a future of increased IT and facility infrastructure modularization to increase redundancy and uptime, he says CoSentry plans to remain “tech-vigilant,” while reducing its impact on the environment.

Fostering a Secure Future
Today’s technology enables the immediate delivery of information, but as we become more accustomed to the convenience provided by technology, so, too, do we become more dependent upon it. “Everything is automated,” states Serbousek. “Records are not on paper anymore.”

Automation has numerous benefits, including environmental advantages, but increased use of technology also imposes demands. “The demand for power is growing,” says Serbousek, noting that more power is used in data centers than in industrial centers. “It’s a fast-growing demand segment of the economy. Congress wants to mandate efficiency, but federal mandates don’t always understand the needs of a data center. Something has to give.”

It better not give at a data center, Dohrmann hopes. “If the power went out, it would be a significant disaster,” he says. “The cost of outage is high: Every minute is millions [of dollars].”

It’s why companies like CoSentry hone their disaster preparedness plans. “You have to have a plan,” he adds.

Although CoSentry prefers to look at prevention instead of recovery, they want to be ready for either scenario, relying on UPS, backed up by generators, backed up by two power grids. Acknowledging the complexity of the plan, Dohrmann says the chance of losing data is remote. However, in order to comply with federal regulations about how fast data has to be made available to the customer, CoSentry isn’t taking any chances. For them, redundancy is a safeguard for resiliency.

“The reserve capacity on the grid is dwindling,” says Serbousek. “The grid is not stable.” That’s why energy-saving features are under the microscope, he says. “It’s tough on coal plants, but wind and alternative energy cost more per kilowatt-hour than coal. Fuel cell, solar array, and wind turbine technology are all in their infancy; it will take 10 years for them to prove themselves. [President] Obama mentioned nuclear energy, which is clean burning and has less impact on the environment, but the market leans toward coal, and unless there are federal mandates, the free market will determine what happens.”

Insisting that it’s possible to blend redundancy with disaster recovery, Serbousek believes onsite power generation provides a vital link in energy-efficient protection.

Author's Bio: Writer Lori Lovely focuses on topics related to transportation and technology.