Maintenance is Risky

No real surprise here. Mission critical facilities that pride themselves on and/or are contractually obligated to provide the “five 9’s” of reliability know that sooner or later they must turn critical cooling equipment off to perform maintenance. And they know that they face risk each time they do so.

This is true even for the newest facilities. The minute a facility is turned up, or IT load is added, things start to change. The minute a brand new cooling unit is deployed, it starts to degrade – however incrementally. And that degree of degradation is different from unit to unit, even when those units are nominally identical.

In a risk and financial performance panel presentation at a recent data center event sponsored by Digital Realty, ebay’s Vice President of Global Foundation Services Dean Nelson recently stated that “touching equipment for maintenance increases Probability of Failure (PoF).” Nelson actively manages and focuses on reducing ebay’s PoF metric throughout the facilities he manages.

Performing maintenance puts most facility managers between the proverbial rock and a hard place. If equipment isn’t maintained, by definition you have a “run to failure” maintenance policy. If you do maintain equipment, you incur risk each time you turn something off. The telecom industry calls this “hands in the network” which they manage as a significant risk factor.

What if maintenance risks could be mitigated? What if you could predict what would happen to the thermal conditions of a room and, even more specifically, what racks or servers could be affected if you took a particular HVAC unit offline?

This ability is available today. It doesn’t require computational fluid dynamics (CFD) or other complicated tools that rely on physical models. It can be accomplished through data and analytics. That is, analytics continually updated by real-time data from sensors instrumented throughout a data center floor. Gartner Research says that hindsight based on historical data, followed by insight based on current trends, drives foresight.

Using predictive analytics, facility managers can also determine exactly which units to maintain and when – in addition to understanding the potential thermal affect that each maintenance action will have on every location in the data center floor.

If this knowledge was easily available, what facility manager wouldn’t choose to take advantage of it before taking a maintenance action? My next blog post will provide a visual example of the analysis facility managers can perform to determine when and where to perform maintenance while simultaneously reducing risk to more critical assets and the floor as a whole.

Predictive Analytics & Data Centers: A Technology Whose Time Has Come

Back in 1993, ASHRAE organized a competition called the “Great Energy Predictor Shootout,” a competition designed to evaluate various analytical methods used to predict energy usage in buildings.  Five of the top six entries used artificial neural networks.  ASHRAE organized a second energy predictor shootout in 1994, and this time the winners included a balance of neural networks and non-linear regression approaches to prediction and machine learning.  And yet, as successful as the case studies were, there was little to no adoption of this compelling technology.

Fast forward to 2014 when Google announced its use of machine learning leveraging neural networks to “optimize data center operations and drive…energy use to new lows.”  Google uses neural networks to predict power usage effectiveness (PUE) as a function of exogenous variables such as outdoor temperature, and operating variables such as pump speed. Microsoft too has stepped up to endorse the significance of machine learning for more effective prediction analysis.  Joseph Sirosh, corporate vice president at Microsoft, says:  “traditional analysis lets you predict the future. Machine learning lets you change the future.”  And this recent article advocates the use of predictive analytics for the power industry.

The Vigilent system also embraces this thinking, and uses machine learning as an integral part of its control software.  Specifically, Vigilent uses continuous machine learning to ensure that predictions driving cooling control decisions remain accurate over time, even as conditions change (see my May 2013 blog for more details).  Vigilent predictive analysis continually informs the software of the likely result of any particular control decision, which in turn allows the software to extinguish hot spots – and most effectively optimize cooling operations with desired parameters to the extent that data center design, layout and physical configuration will allow.

This is where additional analysis tools, such as the Vigilent Influence Map™, become useful.  The Influence Map provides a current, real-time and highly visual display of which cooling units are cooling which parts of the data floor.

As an example, one of our customers saw that he had a hot spot in a particular area that hadn’t been automatically corrected by Vigilent.  He reviewed his Vigilent Influence Map and saw that the three cooling units closest to the hot spot had little or no influence on the hot spot.  The Influence Map showed that cooling units located much farther away were providing some cooling to the problem area.  Armed with this information, he investigated the cooling infrastructure near the hot spot and found that dampers in the supply ductwork from the three closest units were closed.  Opening them resolved the hot spot.  The influence map provided insight that helped an experienced data center professional more quickly identify and resolve his problem and ensure high reliability of the data center.

Operating a data center without predictive analytics is like driving a car facing backwards.  All you can see is where you’ve been and where you are right now.  Driving a car facing backwards is dangerous.   Why would anyone “drive” their data center in this way?

Predictive analytics are available, proven and endorsed by technology’s most respected organizations.  This is a technology whose time has not only come, but is critical to the reliability of increasingly complex data center operations.

IMG_7525_cliff250

A Look at 2013

We grew!

We moved!

We’ve had a heck of a year!

In 2013 alone, we reduced (and avoided the generation of) more than 85 thousand tons of carbon emissions from the atmosphere.

This is a statistic of which I am very, very proud and one that clearly demonstrates the double bottom line impact of the Vigilent solution.

We have directly impacted the planet by reducing energy requirements and CO2 emissions, even as the demands of our digital lifestyles increase.  We have impacted individual quality of life by increasing uptime reliability and contributing to the safety of treasured documents and photos, as well as helping to ensure the uninterrupted transmission of information that makes our world operate.  We are honored and privileged to contribute so directly to the well-being of our world and our customers.

While analysts have cited a DCIM market contraction in 2013, Vigilent has thrived.   We attracted new customers and engendered even deeper loyalty among existing customers – evidenced by our organic growth as one deployment turns into 3, then ten, then dozens across the United States when actual energy savings and thermal condition insights are realized.

I am pleased to share some of the milestones we achieved in 2013:

We moved to terrific new facilities in uptown Oakland.  Not only does our new facility (within a literally green building)  provide us with space for in-house product commissioning and expanded R&D,  it provides a vibrant collaborative atmosphere for employees.  The new location is adjacent to public transportation, honoring our commitment to a green corporate culture, and offers dozens of great restaurants, coffee shops and diverse entertainment options for employees.

We grew – in revenues, in customer base, into new markets and with staff.  With growth comes responsibility to provide more directed  leadership in business functions and market focus.  With this in mind, we expanded our executive management staff, hiring  Dave Hudson to oversee sales and operations worldwide, and  Alex Fielding to introduce Vigilent to federal markets and many new field engineers, software engineers, QA and support staff.

We expanded our product offering with new functionality including out-of-the-box reports that help with energy savings, SLA adherence, maintenance and capacity planning.  We continued to refine our trademark intelligence and control functionality enhancing both usability and energy savings in ever more complex data center environments – achieving an additional 30% savings in some cases.

Ultimately, all of this helps our customers succeed not only in direct bottom line impact, but with large-scale sustainability efforts that are widely recognized.  Avnet used the Vigilent system in corporate sustainability initiatives that garnered the company the Uptime Institute GEIT award, as well as recognition by InfoWorld as a top Green IT award winner.    Our sales partner, NTT Facilities, continues to roll out  Vigilent deployments in Japan.

Our ability to contribute to the Federal Government’s initiative to consolidate data centers and reduce overall energy savings is significant indeed.  Watch this space.

With a great year behind us, we recognize that there is much to do, as the data center industry – at last – is realizing how significantly data and analytics can improve day to day operations and efficiency endeavors.

The Emerson-Poneman Institute recently issued a study on Data Center outages that states accidental human error remains in the top-3 cited reasons for downtime and that 52% of survey respondents believe these accidents could have been prevented.

Intelligent software control and analytics will help operators make better,  more informed decisions and reduce such human errors.   These tools will increasingly help data centers proactively avoid trouble, while at the same time helping them diagnose and resolve actual issues more quickly.

This will be the year of analytics for data centers.  Vigilent is equipped and prepared to lead this charge, leveraging years of institutional knowledge we have gleaned  from hundreds of deployments in every conceivable configuration in mission critical facilities on four continents.  This mass of data influences the analytics we use to engage individual control decisions at every site, and also, more recently, places the benefit of this accumulated knowledge into the hands and minds of data center managers for more informed process management.

Happy New Year.

The Value of Efficiency-Aware Decision Making

My Chevy Volt displays my gas mileage.  In fact, I knew what the mileage performance would be before I bought the car. It was a factor in my purchase choice.

In addition to cars, most large appliances display power use along with Energy-Star certification. Residential air conditioners display standard energy efficiency ratings (SEER).   Even large commercial building air conditioners have to meet standard rating conditions for efficiency.

Yet, it is only recently that efficiency ratings have been specified for data center cooling.  The primary reason is that for years, manufacturers of cooling units for mission critical facilities avoided efficiency ratings requirements claiming that, because their products were used for process cooling versus comfort cooling, efficiency standards shouldn’t apply.  Fortunately, ASHRAE took up the charge and updated Standard 90.1 so that equipment covered by ASHRAE Standard 127 is required to meet minimum efficiency standards.  Standard 90.1 has been adopted by the Department of Energy as a federal energy standard and is now referenced by many code authorities.

While useful and certainly progress, the choice enabled by these two standards is just a start.  Certainly new equipment can and should be compared based on energy efficiency ratings.  However we all know that equipment efficiency will vary considerably through use. It would also be useful to be able to  view and compare the operational efficiency of existing equipment in order  to evaluate which machines are working well, which should be replaced (using the new equipment efficiency ratings as a baseline of comparison) –  and how much efficiency could be gained (and calculated from an ROI perspective) through replacement.

Some HVAC manufacturers have taken up this challenge. NTT, for example, provides the coefficient of performance for its computer room air conditioners in real time, viewable on the front panel of each unit and through a communications interface.  We commend them.

The ability to compare initial purchase energy efficiency ratings against actual performance over time for a particular machine, gives data center managers the ability to not only track and evaluate a machine for individual performance durability, and compare its performance with that of similar machines.  Mechanisms and procedures can be put in place for maintenance as degradation is spotted.   Inefficient machines can be used less, fixed or phased out.

We challenge mission critical cooling system manufacturers to pull back the veil of secrecy on energy efficiency.  The time for transparency is at hand because this information is knowable.  The combination of smart sensors and analytics technology can already report dynamic machine-to-machine efficiency as this information is required to drive cooling optimization.  The smart decision is for HVAC manufacturers to get out ahead of this data, and use efficiency reporting as a differentiator and means of driving continual improvement.

Just as mandatory EPA mileage ratings and rising gas prices changed consumer buying decisions – and drove car manufacturers to offer cars with better gas mileage, more granular energy performance ratings will improve the efficiency of cooling equipment.  And this benefits all of us.

Intelligent Efficiency

Intelligent Efficiency, The Next New Thing.

Greentech Media’s senior editor Stephen Lacey reported that the convergence of the internet and distributed energy are contributing to a new economic paradigm for the 21st century.

Intelligent efficiency is the next new thing enabled by that paradigm, he says, in a special report  of the same name.  He also notes that this isn’t the “stale, conservation-based energy efficiency Americans often think about.”  He says that the new thinking around energy efficiency is information-driven.  It is granular. And it empowers consumers and businesses to turn energy from a cost into an asset.

I couldn’t agree more.

Consider how this contrast in thinking alone generates possibilities for resources that have been hidden or economically unavailable until now.

Conservation-based thinking or, as I think about it in data centers, “efficiency by design or replacement,” is capital intensive.  To date, this thinking has been focused on new construction, physical infrastructure change, or equipment swap-outs.  These efforts are slow and can’t take advantage of operational variations such as the time-varying costs of energy.

Intelligent energy efficiency thinking, on the other hand, leverages newly available information enabled by networked devices and wireless sensors  to make changes primarily through software.  Intelligent energy management is non-disruptive and easier to implement.  It reduces risk by offering greater transparency.   And, most importantly, it is fast.  Obstacles to the speed of implementation – and the welcome results of improved efficiency – have been removed by technology.

Intelligence is the key factor here.  You can have an efficient system, an efficient design, but if it isn’t operated effectively, it is inherently inefficient.  For example, you may deploy one perfectly efficient machine right next to another perfectly efficient machine believing that you have installed a state-of-the-art solution.  In reality, it’s more likely that these two machines are interacting and fighting with each other – at significant energy cost.   You also need to factor in and be able to track equipment degradation as well as the risks incurred by equipment swap-outs.

You need the third element – intelligence – working in tandem with efficient equipment, to make sure that the whole system works at peak level and continues to work at peak level, regardless of the operating conditions.  This information flow must be constant.  Even the newest, most perfectly optimized data centers will inevitably change.

Kudos to Greentech Media for this outstanding white paper and for highlighting how this new thinking and the” blending of real-time communications with physical systems”  is changing the game for energy efficiency.

2012 Retrospective

It’s getting better all the time.

Despite our relentless drive to consume more and more data, driven by ever more interesting and arguably useful multimedia applications, energy consumption of data centers is growing slower than would be predicted from historical trends.

For that success, we should be proud, while remaining focused on even greater efficiency innovation.

Large companies have stepped up with powerful sustainability initiatives which impact energy use throughout their enterprise. We’ve gotten better at leveraging natural resources, like outside air to moderate data center temperatures.  We are using denser, smarter racks for space and other efficiencies. Data center cooling units are built with variable speed devices improving energy efficiency machine-by-machine. Utility companies are increasingly offering sophisticated and results-generating incentives to jump-start efficiency programs.

These and other contributing factors are making a difference, clearly proven in Jonathan Koomey’s Growth in Data Center Electricity Use 2011 report which showed a flattening, versus a lockstep correlation of energy usage to data center growth. Koomey and other analyst growth estimates projected a doubling of world data center energy usage from 2005 to 2010.  Actual growth rates were closer to 56%, a reduction that Koomey attributes both to fewer than expected server installations – and a reduced use of electricity per server.

I am proud of what our industry – and what our company – has achieved.  Consider some of this year’s highlights.

The New York Times raised the profile – and the ire  – of the data center industry calling attention to the massive energy consumed by, well, consumers.  Data center facilities and analysts alike responded with criticism, saying that the article ignored the many and significant sustainability and energy use reductions now actively in use.

Vigilent received an astounding 8 industry awards this year – recognizing our technology innovation, business success and workplace values. I’m very proud of the fact that several of these awards were presented by or achieved in partnership with our customers.  For example, Vigilent and NTT won the prestigious Uptime GEIT 2012 award in the Facility Product Deployment Category.  NTT Facilities with NTT Communications received the 2012 Green Grid Grand Prix award, recognizing NTT’s innovative efforts in raising the energy efficient levels of Japan by using Vigilent and contributing DCIM tools.  And Verizon, in recognition of our support for their commitment to continuing quality and service, presented us with their Supplier Recognition award in the green and sustainability category.

We moved strongly into Japanese and Canadian markets with the help of NTT Facilities and Telus, both of whom made strategic investments in Vigilent following highly successful deployments.  Premiere Silicon Valley venture firm Accel Partners became an investor early in the year.

We launched Version 5 of our intelligent energy management system adding enhanced cooling system control with Intelligent Analytics-driven trending and visualization, along with a new alarm and notification product to further reduce downtime risk.

And, perhaps most satisfyingly of all, we helped our customers avert more than a few data center failures through real-time monitoring and intercession, along with early notification of possible issues.

This year, we will reduce energy consumption by more than 72 million kWh in the US alone.  And this figure grows with each new deployment.  We do this profitably, and with direct contribution to our customer’s bottom line as well through energy cost savings.

Things are getting better. And we’re just getting started.