The Real Cost of Cooling Configuration Errors

Hands in the network cause problems. A setting adjusted once, based on someone’s instinct of what needed to be changed at one moment in time, is often unmodified years later.

This is configuration rot. If your data center has been running for a while, the chances are pretty high that your cooling configurations, to name one example, are wildly out of sync. It’s even more likely you don’t know about it.

Every air conditioner is controlled by an embedded computer. Each computer supports multiple configuration parameters. Each of these different configurations can be perfectly acceptable. But a roomful of air conditioners with individually sensible configurations can produce bad outcomes when their collective impact is considered.

I recently toured a new data center in which each air conditioner supported 17 configuration parameters affecting temperature and humidity. There was a lot of unexplainable variation in the configurations. Six of the 17 configuration settings varied by more than 30%, unit to unit. Only five configurations were the same. Configuration variation initially and entropy over time wastes energy and prevents the overall air conditioning system from producing an acceptable temperature and humidity distribution.

Configuration errors contribute to accidental de-rating and loss of capacity. This wastes energy, and it’s costly from a capex perceptive. Perhaps you don’t need a new air conditioner. Instead, perhaps you can optimize or synchronize the configurations for the air conditioners you already have and unlock the capacity you need. Another common misconfiguration error is incompatible set points. If one air conditioner is trying to make a room cold and another is trying to make it warmer, the units will fight.

Configuration errors also contribute to poor free cooling performance. Misconfiguration can lock out free cooling in many ways.

The problem is significant. Large organizations use thousands of air conditioners. Manual management of individual configurations is impossible. Do the math. If you have 2000 air conditioners, each of which has up to 17 configuration parameters, you have 34,000 configuration possibilities, not to mention the additional external variables. How can you manage, much less optimize configurations over time?

Ideally, you need intelligent software that manages these configurations automatically. You need templates that prescribe optimized configuration. You need visibility to determine, on a regular basis, which configurations are necessary as conditions change. You need exception handling, so you can temporarily change configurations when you perform tasks such as maintenance, equipment swaps, and new customer additions, and then make sure the configurations return to their optimized state afterward. And, you need a system that will alert you when someone tries to change a configuration, and/or enforce optimized configurations automatically.

This concept isn’t new. It’s just rarely done. But if you aren’t aggressively managing configurations, you are losing money.

When Free Cooling Isn’t Free

Published in Data Center Dynamics.

The use of free cooling systems is quickly becoming common practice – particularly in new mission critical facility builds. Using outside air, either directly or indirectly, to cool ICT equipment is undeniably compelling, both logically and financially.

But is free air really free? Not always. Free cooling systems add considerable complexity to the operation and maintenance of mechanical equipment. If this complexity isn’t recognized or managed well, free cooling will add to energy costs and increase operational risk.

Watch the weather

Weather is the most obvious variable. Free cooling capacity declines in hot weather, requiring a design that either allows for elevated indoor temperatures or combines free cooling with conventional mechanical cooling to ensure that indoor temperatures remain within an acceptable range.

Multiple operating modes are another complicating factor. For example, the free cooling system at Facebook’s Prineville data center (pictured) uses eight distinct operating conditions to optimize the use of direct outside air and direct evaporative cooling under different weather conditions. Free cooling systems that use direct outside air augmented by compressorized cooling have at least three distinct operating conditions.

Maintenance also becomes more complex. Free cooling adds to the number of moving mechanical components (e.g. air dampers and actuators) that are in direct contact with outdoor air. Outdoor air is corrosive, which can cause the dampers and actuators to get stuck, and either fail to provide cooling or cause the system to bring in hot outdoor air when it should not. Free cooling systems with evaporative cooling have the added maintenance of cooling water, which requires chemical treatment and periodic flushing.

This complexity can significantly impact the energy reduction that free cooling can deliver, while creating real thermal management problems.

High failure rates

Accordingly, the high failure rates of free cooling systems are well documented in energy efficiency and building technology literature. A particularly good and practical paper entitled Free Cooling, At What Cost was written by Kristen Heinemeier and presented at the 2014 ACEEE Summer Study on Energy Efficiency in Buildings. My direct experience with free cooling systems throughout the US and Europe is completely consistent with Heinemeier’s paper. Specifically, I have seen even higher failure rates in mission critical facilities than in the commercial buildings referenced in Heinemeier’s paper.

Heinemeier examined the prevalence and impact of air-side economizer (direct free cooling) failure. She found that although economizers are an excellent energy saving technology, they do not perform well in practice. In California alone, she cites that in surveyed facilities, the economizer is disabled and outside air dampers are closed 30 – 40 percent of the time. She states: “This type of failure means that the economizer is not providing any savings, and that the building may not be bringing in any outside air. Other studies have found that the high-limit setpoints, set by technicians, are incorrect on the majority of RTUs in California, resulting in very few hours in the ‘free cooling’ range.”

I recently toured five sites in two countries, owned by different multinational companies, using cooling equipment from three different manufacturers.

Among the dozens of free cooling units that I observed on this trip, nearly all either had a problem that limited capacity and function or weren’t working at all. Problems included controller configuration, sensor failure, installation faults, and mechanical failures.

Some examples:

  • In one site, the outdoor air was cool but the outside air dampers were fully closed and the unit was recirculating indoor air. The temperature remained within an acceptable range; however, this was because the DX compressors were running unnecessarily – at massive cost. The operators knew that the free cooling should be operating, but didn’t know why it wasn’t. The facility had been operating that way since the free cooling units had been installed – about a year prior. Inspection of the units revealed that the controls weren’t configured properly, and that misconfigured control logic was preventing the free cooling from operating. I saw a similar scenario in a second site.
  • At another site I observed that the controls were working and appeared to be pulling in outside air. However, the discharge air on one particular unit wasn’t as cold as I would have expected. Inspection of the unit revealed that BOTH the outside air dampers and the return air dampers were closed. The damper actuator clamp on the outside air damper had either fallen off or been removed, leaving that damper stuck in the fully closed position. This problem was identified by analyzing data from the cooling optimization sensor network
  • At yet another site, I saw that the controls were working, the dampers were working and that cold air was produced – just not very much. We measured a large temperature difference in the outdoor air intake across the outside wall. The outside air duct was installed with a flanged connection to the wall. At a nearby site with the same free cooling equipment, the outside air duct penetrated the wall. The flanged installation caused the cooling units to draw air from the hollow wall construction, reducing the capacity of the free cooling by up to 40 percent. This problem was also identified by analyzing sensor network data.

What’s important to note is that while in each case the free cooling system had problems, they were all fixable problems – often with little or no investment. More significantly, operators didn’t always recognize that their free cooling was compromised, nor how it could be fixed. Besides the additional energy costs and potential thermal risk incurred by this lack of visibility, these facilities were on the verge of spending a lot of money in pursuit of a solution, when in fact their existing equipment would achieve the desired operation.

Monitor your cooling system

Because free cooling systems are highly efficient when they do work as intended, best practice would suggest that risk mitigation and visibility through a monitoring system is required to realize the safe operation and full benefit of free cooling. In California, Title 24 requires diagnostics for use with free cooling systems. Dynamic monitoring, analytics, and diagnostics in conjunction with visual inspection will reveal issues and help ensure the ongoing and proper operation of free cooling within a complex cooling infrastructure. In mission critical facilities that are operated lights-out, use of remote monitoring and analytics combined with intelligent alerting is the only way to ensure reliable operation of free cooling.

As free cooling becomes a standard means of cooling mission critical facilities, consideration of the risk and complexity it adds is critical. Data-driven oversight of cooling operations, in combination with a layer of smart analytics and control, is the best-practice way to ensure your thermal environment continually operates in the most efficient way possible. This oversight also ensures that you continue to optimize your capital investment, even as conditions, weather and physical changes occur over time.