Right Hosting Outtage this morning.
 

To all that were affected this morning, my sincerest apologies.

Our building management scheduled an air conditioning up grade this morning to double our AC capacity. Hosting is a balance of many factors, cool air is a big part of that equation. Having done some significant planning for this upgrade (extra staff, fans etc) we felt prepared though we had been promised a Zero downtime upgrade.

That being said, The building's AC people Disconnected Every unit, rather than one at a time. They brought large temporary units to temporarily supplant the one unit, but not enough to replace 60 plus tons of AC.

IN the wonder of the full shut down, water to all units was also disconnected, another important part of the balance, The contractors were unaware of the problem they were creating as they did this. What this did was shutdown all AC to the 6,000 plus square feet to our floor of the UUNet building we reside in.

Servers began to overheat, in so doing they raise their power consumption rapidly often doubling or tripling that consumption. The effect of this is that some of our power circuits blew. This is what caused servers to go down.

We have spent the morning re-routing power, adding temp AC Units, bringing the other units back online and slowly bring up machines as we cool the data center and route power to all cabinets and racks.

I am truly sorry for the trouble the buildings contractors have caused us and thus to you, our clients. feel free to call me directly if you have any questions.

Caz Thrush
CTO
818-636-6710

Glad to hear the issue is getting resolved Caz :thumbsup

So where were you with your hand fan????


:)

OK. Now show me your lowest price for hosting ICQ me when you have time. And I want the We Broke our AC special hosting deal. :)


No, Reall. ICQ me when you can or one of your guys.

I would have strangled those damn contractors, they should have known they couldnt shut off the water from the chillers to the krac units. Im suprised there werent any server meltdowns or fires from the massive power surge from overheating

Like SinisterStudios said above Won't overheating a server to the point of raising it's power that high cause horrific damage to it and its components like hard drives, mobo's power supplies ect...? hummmmm doesn't sound to kosher to me.:2 cents:

it was close we had to do some overheat shutdowns, while they brought the old and temp units online

Man, how do "contractors" get inside a secured building and just go ahead and fucking turn off everything without anyone watching? I'm ALWAYS amazed that this continues to happen at every datacenter or host....

Is there no foresight or is this the usual case of everyone thinking that the next guys got and eye on it so I don't have to worry, and no ones got an eye on anything.

Great job to all of the Righthosting team for working so hard to get things resolved.

Any truth to the rumor that Lisa Sparxxx was behind the whole thing?