Originally Posted by Techie Media

The problem started at 10am EST when we immediately noticed that our links were down to our second datacenter. We saw link but no activity on our 2 sets of fiber that connect our 5th floor datacenter with our 8th floor datacenter.

We run a layer-3 switch setup with load balanced hot spare connected to our core network between floors to light our second facility. Our immediate thought was that the primary switch failed in some way and the secondary did not take over like it should. This lead to a very long trouble shooting process with our tech even involving the switch vendor who had us update IOS and just about every test they could thing of with no result since the switches were identical in every way we had to be sure. We then decided to put a cold spare switch online to take their place which was a different brand and we had the same results, link and no activity.

The next step was to have the fiber company that handles our dark fiber test everything and also the the building meet me room check everything, they both told us the problem must be with our equipment.

We went back to more trouble shooting and actually brought the primary switch from the second datacenter all the way up into the main datacenter and connected it to the core on short fiber and it was working fine. It just did not work when connected between floors.

We then went back and demanded that the fiber company take a second look. This time they DID find and issue and corrected it and our second datacenter was back online.

This is one of those times we did everything we possibly could, worked tirelessly with all vendors that were involved and the problem was still caused and then prolonged by something we could not control as the hours of the day just seemed to fly by. So, to sum it up we got hit in a soft spot and it hurt.

We have been in constant contact with our customer and many have been told what we just posted by phone and ICQ.

The situation has been 100% resolved.