|
Just to give you guys a timeline of events here ?
Saturday/Sunday:
We offered our assistance to RCN, free of charge, with no conditions. We assisted them in doing forensic analysis of their servers in one of our facilities. At the end of the day, we had both come to the conclusion that it would be impossible to recover directly from the hard drives as both primary and backup Supernodes, as well as the partition table had been overwritten.
At this point, we assisted RCN in re-gaining access to their equipment by installing operating systems onto the boot drives of hundreds of servers ? all in a single day. This was done without any cost, and without any conditions.
Monday:
RCN had made the determination (which they later changed), that they would be unable to recover their customers in a reasonable amount of time, and asked that we begin to offer a migration plan to their customers to our network. RCN had said at this point that they would offer assistance in this transition process, which ultimately never came.
Tuesday:
RCN had decided that they will try to save their business, and they are now working independently to recover customers. I do not have any idea of the progress there.
And some behind-the-scenes facts:
This recovery requires a lot of staff working overtime, some staff from other regions to be relocated (requiring travel and lodging), as well as staff pulled in from vacations, and large amounts of new equipment, delivered ?ASAP?. We do maintain a very large inventory, but quite obviously, we have had to make some replenishments at less-than-ideal pricepoints. We are not passing on any of these excessive costs on to customers, but we cannot simply wave a magic wand and make 1000 servers appear for free.
Our support staff is working tirelessly to try and clean up this mess, and have the following considerations:
1) Time spent talking to customers is directly taking away from time spent recovering customer data ? it?s our priority to get customers recovered as quickly as possible.
2) Customers have in many cases not provided large amounts of critical information relating to their setup. All information that was relayed was properly handled, however ultimately customers are not satisfied that what they didn?t tell us was not handled. We are not mind readers, and we are taking on new customers who we do not have a prior knowledge of their setups.
3) Customers are very understandably highly stressed, saddened, overwhelmed, etc. We understand this. However, customers calling up our support staff and (in some cases, quite literally) crying is both very stressful for our staff and takes up valuable time.
4) On top of all other constraints here, The backup servers are highly overloaded at this point ? and again, I don?t think anyone ever considered the concept of having to restore 1000 servers simultaneously, and are in many cases, frankly very lucky that a backup exists in the first place, as most users were not even paying for backup.
5) This is not like moving an existing customer elsewhere who has working hosting -- we are recovering from a non-working backup. This is actually much more work than a typical "new customer setup", and again, we are not charging any extra fees for this.
Bottom line ? as was said by someone else, this is a business, so there are some ?real world? considerations, but we are genuinely trying to help. We did not ask for this situation, and wouldn?t wish this on anyone. Please have some understanding and work with us. Trashing us on forums, or yelling at our staff is not helping anyone get their data back. If you are a customer and need help, please email us, with all relevant information, and be understanding of the fact that there are no immediate or easy solutions here.
__________________
|