Big Ray |
01-10-2004 08:48 PM |
Hey Folks.
Ray here from Jupiter.
We had an outage tonight.
Started: 01-10-2004 17:58:36
Ended: 01-10-2004 19:04:44
A Few Facts:
1) This outage affected ¼ our our customer base. As it was already stated, some sites were up, some were down. If your sites are still experiencing a problem please send a ticket in.
2) The cause was not due to DNS or Fiber. The cause was a failure in a primary route / switch card in our BigIron 8000. Unfortunately, it was not a hard down situation (from a switch stand point). Functionality was spotty and the redundant management card did not take over (as it had not detected an outage on the primary.)
3) Our on-site techs did see the problem and was alerted to the outage seconds after it happened. Seconds after that, customers obviously started calling. It so happens that the tech that picked up Shap?s call was in an overflow tech queue and is home sick. His phone only rings when all other techs are on the phone (emergency situation). As he was not on duty he had not been notified of the outage.
4) Our onsite techs were able to trace and isolate the problem quickly. We have onsite sparing of line cards for our switches, so swapping out the bad card and reloading the router config into flash from CVS was relatively simple. The switch is back online.
5) Customers effected by the earlier 22 minute outage this week as well as this one will be receiving a 10% discount on hosting for the month. Please advise if this is not reflected in your EOM invoice.
I would also like to state that for the last 12 months on this network, we have had a total of 78 minutes of downtime effecting a percentage of customers. We have not had a single outage that affected all customers. As we grow and make our network more resilient, we expect this 78 minutes of downtime to decrease even further.
With that said, please accept our apologies.
Thanks for your cooperation and patience. I will be on ICQ at 74335014 if you wish to chat.
Ray
|