Right Hosting Outtage this morning. - GoFuckYourself.com

Ycaza · 05-28-2009, 11:34 AM

To all that were affected this morning, my sincerest apologies.

Our building management scheduled an air conditioning up grade this morning to double our AC capacity. Hosting is a balance of many factors, cool air is a big part of that equation. Having done some significant planning for this upgrade (extra staff, fans etc) we felt prepared though we had been promised a Zero downtime upgrade.

That being said, The building's AC people Disconnected Every unit, rather than one at a time. They brought large temporary units to temporarily supplant the one unit, but not enough to replace 60 plus tons of AC.

IN the wonder of the full shut down, water to all units was also disconnected, another important part of the balance, The contractors were unaware of the problem they were creating as they did this. What this did was shutdown all AC to the 6,000 plus square feet to our floor of the UUNet building we reside in.

Servers began to overheat, in so doing they raise their power consumption rapidly often doubling or tripling that consumption. The effect of this is that some of our power circuits blew. This is what caused servers to go down.

We have spent the morning re-routing power, adding temp AC Units, bringing the other units back online and slowly bring up machines as we cool the data center and route power to all cabinets and racks.

I am truly sorry for the trouble the buildings contractors have caused us and thus to you, our clients. feel free to call me directly if you have any questions.

Caz Thrush
CTO
818-636-6710

Ice · 05-28-2009, 12:00 PM

Glad to hear the issue is getting resolved Caz

pornguy · 05-28-2009, 12:51 PM

So where were you with your hand fan????

OK. Now show me your lowest price for hosting ICQ me when you have time. And I want the We Broke our AC special hosting deal.

No, Reall. ICQ me when you can or one of your guys.

SinisterStudios · 05-28-2009, 01:29 PM

I would have strangled those damn contractors, they should have known they couldnt shut off the water from the chillers to the krac units. Im suprised there werent any server meltdowns or fires from the massive power surge from overheating

Net Money · 05-28-2009, 09:24 PM

Like SinisterStudios said above Won't overheating a server to the point of raising it's power that high cause horrific damage to it and its components like hard drives, mobo's power supplies ect...? hummmmm doesn't sound to kosher to me.

Ycaza · 05-28-2009, 09:57 PM

Quote:

Originally Posted by SinisterStudios

I would have strangled those damn contractors, they should have known they couldnt shut off the water from the chillers to the krac units. Im suprised there werent any server meltdowns or fires from the massive power surge from overheating

it was close we had to do some overheat shutdowns, while they brought the old and temp units online

Iron Fist · 05-28-2009, 10:17 PM

Man, how do "contractors" get inside a secured building and just go ahead and fucking turn off everything without anyone watching? I'm ALWAYS amazed that this continues to happen at every datacenter or host....

Is there no foresight or is this the usual case of everyone thinking that the next guys got and eye on it so I don't have to worry, and no ones got an eye on anything.

NETbilling · 05-28-2009, 10:49 PM

Great job to all of the Righthosting team for working so hard to get things resolved.

Snake Doctor · 05-29-2009, 10:40 AM

Any truth to the rumor that Lisa Sparxxx was behind the whole thing?

05-28-2009, 11:34 AM	#1
Ycaza Confirmed User Industry Role: Join Date: Feb 2005 Location: the 805 Posts: 4,290	Right Hosting Outtage this morning. To all that were affected this morning, my sincerest apologies. Our building management scheduled an air conditioning up grade this morning to double our AC capacity. Hosting is a balance of many factors, cool air is a big part of that equation. Having done some significant planning for this upgrade (extra staff, fans etc) we felt prepared though we had been promised a Zero downtime upgrade. That being said, The building's AC people Disconnected Every unit, rather than one at a time. They brought large temporary units to temporarily supplant the one unit, but not enough to replace 60 plus tons of AC. IN the wonder of the full shut down, water to all units was also disconnected, another important part of the balance, The contractors were unaware of the problem they were creating as they did this. What this did was shutdown all AC to the 6,000 plus square feet to our floor of the UUNet building we reside in. Servers began to overheat, in so doing they raise their power consumption rapidly often doubling or tripling that consumption. The effect of this is that some of our power circuits blew. This is what caused servers to go down. We have spent the morning re-routing power, adding temp AC Units, bringing the other units back online and slowly bring up machines as we cool the data center and route power to all cabinets and racks. I am truly sorry for the trouble the buildings contractors have caused us and thus to you, our clients. feel free to call me directly if you have any questions. Caz Thrush CTO 818-636-6710 __________________ Caz Thrush Head Honcho [email protected] http://thrushtech.com ICQ: 304883574 do people still icq?

05-28-2009, 12:00 PM	#2
Ice Confirmed User Join Date: Nov 2002 Location: Sunny California Posts: 26,053	Glad to hear the issue is getting resolved Caz __________________ icq 1904905

05-28-2009, 12:51 PM	#3
pornguy Too lazy to set a custom title Industry Role: Join Date: Mar 2003 Location: Homeless Posts: 62,912	So where were you with your hand fan???? OK. Now show me your lowest price for hosting ICQ me when you have time. And I want the We Broke our AC special hosting deal. No, Reall. ICQ me when you can or one of your guys. __________________ PornGuy skype me pornguy_epic AmateurDough The Hottes Shemales online! TChicks.com \| Angeles Cid \| Mariana Cordoba \| MAILERS WELCOME!

05-28-2009, 01:29 PM	#4
SinisterStudios Confirmed User Join Date: Nov 2003 Location: New Joisey Posts: 3,087	I would have strangled those damn contractors, they should have known they couldnt shut off the water from the chillers to the krac units. Im suprised there werent any server meltdowns or fires from the massive power surge from overheating __________________ SEOIP.com Multiple IP Webhosting Shared and Dedicated IP's - Multiple Class A's - From $1.99/ip

05-28-2009, 09:24 PM	#5
Net Money Confirmed User Join Date: Dec 2007 Posts: 539	Like SinisterStudios said above Won't overheating a server to the point of raising it's power that high cause horrific damage to it and its components like hard drives, mobo's power supplies ect...? hummmmm doesn't sound to kosher to me. Last edited by Net Money; 05-28-2009 at 09:25 PM..

05-28-2009, 10:17 PM	#7
Iron Fist Too lazy to set a custom title Join Date: Dec 2006 Posts: 23,400	Man, how do "contractors" get inside a secured building and just go ahead and fucking turn off everything without anyone watching? I'm ALWAYS amazed that this continues to happen at every datacenter or host.... Is there no foresight or is this the usual case of everyone thinking that the next guys got and eye on it so I don't have to worry, and no ones got an eye on anything. __________________ i like waffles

05-28-2009, 10:49 PM	#8
NETbilling Confirmed User Industry Role: Join Date: Jan 2002 Location: Huntington Beach, CA Posts: 8,573	Great job to all of the Righthosting team for working so hard to get things resolved. __________________ Mitch Farber CEO - NETbilling, Inc. Email / Phone: 888-357-8166 / 661-252-2456 Transaction processing & 24/7 call center services with exceptional rates and flexibility, since 1998!

05-29-2009, 10:40 AM	#9
Snake Doctor I'm Lenny2 Bitch Join Date: Mar 2001 Location: On top of my soapbox Posts: 13,449	Any truth to the rumor that Lisa Sparxxx was behind the whole thing? __________________ sig too big