|
Right Hosting Outtage this morning.
To all that were affected this morning, my sincerest apologies.
Our building management scheduled an air conditioning up grade this morning to double our AC capacity. Hosting is a balance of many factors, cool air is a big part of that equation. Having done some significant planning for this upgrade (extra staff, fans etc) we felt prepared though we had been promised a Zero downtime upgrade.
That being said, The building's AC people Disconnected Every unit, rather than one at a time. They brought large temporary units to temporarily supplant the one unit, but not enough to replace 60 plus tons of AC.
IN the wonder of the full shut down, water to all units was also disconnected, another important part of the balance, The contractors were unaware of the problem they were creating as they did this. What this did was shutdown all AC to the 6,000 plus square feet to our floor of the UUNet building we reside in.
Servers began to overheat, in so doing they raise their power consumption rapidly often doubling or tripling that consumption. The effect of this is that some of our power circuits blew. This is what caused servers to go down.
We have spent the morning re-routing power, adding temp AC Units, bringing the other units back online and slowly bring up machines as we cool the data center and route power to all cabinets and racks.
I am truly sorry for the trouble the buildings contractors have caused us and thus to you, our clients. feel free to call me directly if you have any questions.
Caz Thrush
CTO
818-636-6710
|