Originally Posted by Techie Media

The problem started at 10am EST when we immediately noticed that our links were down to our second datacenter. We saw link but no activity on our 2 sets of fiber that connect our 5th floor datacenter with our 8th floor datacenter.

We run a layer-3 switch setup with load balanced hot spare connected to our core network between floors to light our second facility. Our immediate thought was that the primary switch failed in some way and the secondary did not take over like it should. This lead to a very long trouble shooting process with our tech even involving the switch vendor who had us update IOS and just about every test they could thing of with no result since the switches were identical in every way we had to be sure. We then decided to put a cold spare switch online to take their place which was a different brand and we had the same results, link and no activity.

The next step was to have the fiber company that handles our dark fiber test everything and also the the building meet me room check everything, they both told us the problem must be with our equipment.

We went back to more trouble shooting and actually brought the primary switch from the second datacenter all the way up into the main datacenter and connected it to the core on short fiber and it was working fine. It just did not work when connected between floors.

We then went back and demanded that the fiber company take a second look. This time they DID find and issue and corrected it and our second datacenter was back online.

This is one of those times we did everything we possibly could, worked tirelessly with all vendors that were involved and the problem was still caused and then prolonged by something we could not control as the hours of the day just seemed to fly by. So, to sum it up we got hit in a soft spot and it hurt.

We have been in constant contact with our customer and many have been told what we just posted by phone and ICQ.

The situation has been 100% resolved.

Originally Posted by daizzzy

there's no reports at webhosting talk atm

Originally Posted by rowan

Just sidetracking for a moment, and this has nothing specifically to do with Techiemedia...

How many of you have complete backups - or at least enough to get you going again - to recover from a catastrophic event such as fire physically destroying your server, or an extended outage?

Or something similarly devastating such as a host quietly going out of biz, and by the time you hear of it your server has been trucked off to who knows where to have its HD erased and given to a new customer?

(And if you have no backups - then something as simple as a HD failure could completely wipe you out...)

Just something to think about.