so you lied when you said it was an ex-employee?

"As for the intrusion it was the result of an ex-employee who was with us for three years as a result he had intimate knowledge of our systems which is why the effects are so large."

Wow......

Probably the biggest hack in history!

Has something like this ever happened before?

RCN have finally copied my sql dumps to my new server. Thankyou RCN. Now where are those red bulls..

Downtime is one thing - paying for a backup service you can't access to right these wrongs is totally another...

the official email says all backup data has been transferred over to isprime - so the onus is with them....

please get me access to this backup data so I can move it to another box to get sites back online - this is what a backup service was paid for... to put right a wrong quickly and efficiently.

I hope ISPrime can show their mettle.

If you run a 1000 server facility then shouldn't you also have a backup system to cope with , well, the destruction of the facility ?

I called ISPrime 5 times about this, their stance is they are helping out with customers migrating to ISPrime only. They have access to the backups for this purpose. If you are migrating to any other network you need to contact RCN. God speed to them.

I really hope the person responsible for this attack is brought to justice.

What a horrible situation!

If anybody needs help in getting their hardware back, I am located in New Jersey and can mail them out to you.

Yeah I noticed that as well - the story has gone from ex employee to hackers, with an s at the end. Are you sure you're not just making this up as you go along?

Perhaps you might want to post this information on your website as well. I'm sure you must have customers that don't get there news from adult message boards. And your website hasn't changed its message since it came back up.

http://www.realitychecknetwork.com/

Based on our experience, the "it would take over a month to restore everything" line is not
is strange as it sounds. A lot of people still use magnetic TAPE to back up. As a 1954 technology,
backup tapes can restore about 1 MB / second, or 86 GB / day. Only 2,600 GB would take a
month.
That's best case perfromance assuming the tech is sitting there the whole time watching closely
and immediately switching tapes and such. A mistake requiring a "do over" doubles the time,
of course.
So it's not unusual for restores to take a long time. I completely believe that, since most backup
systems I've seen have huge problems, and being super slow is a very common problem.

Personally, I'm not fond of living in 1954, so for Clonebox we use 16 drive wide RAID arrays
capable of pushing about 100 times that much throughput. If it would take a month to restore
all of the servers from tape, Clonebox could do it in about eight hours, or the sites could simply
run from the Clonebox servers, which takes only a few minutes to set up.

Acts of god aren't covered by your insurance either, this being the equivalent of that. Be grateful the backups weren't corrupted.

True that. However if you run a large facility then having a tape only backup system that can only cope with a few restores at a time is tantamount to not having a backup because the point of a backup is that you can deploy a restore quickly if need be.

Disaster recovery plans should cater for the worst case scenarios and obviously RCN, for all the hype of their managed server operations, didn't put enough work into developing an effective disaster recovery plan.

Now it seems they are pulling the plug on all these servers and forgoing the leases on them while giving ISPrime access to the backups. If what they say is true then regardless of who does the work, 4000 man hours of work is required to complete a restore. That is disastrous.

If you are running mission critical applications then you should insist that your host provides rotating offsite disk based backups so that disaster recovery is possible, because no disaster recovery plan is successful if it takes a month. :2 cents:

Note to self : Hackers have finally attained the status of God.

:1orglaugh

If a couple of days of downtime loses you $60k why don't you have a duplicated server (or servers) at another host?

Given the low cost of servers these days, you'd think this would be obvious wouldn't you ?

This is a good time for everyone to reflect on what would happen if their host went up in a puff of smoke and take steps to ensure they have some redundancy in their operation at all mission critical levels.

its not lost sales, its a lost database with 60k in rebills. And yes i do have backups of everythign 10 times over but not the userbase which they told me was being backed up with double redundancy.

Regardless, i finally made the proper threats and talked to the right people and got my files. so I am fine now. they were making up the proper backups all along, i just never got the right person on the horn.

Comment still stands: you should be making an offsite backup of that very valuable database at the least, even to the HD on your home computer.

Glad you got your data back, now consider taking better precautions going forward. :thumbsup

Personally, I wouldn't trust my business to anyone who's own website doesn't come up when you google their entire name.

Being a vulture wont likely net you any clients and implying that any serious hosts still uses tape storage, namely isprime, is an insult to your intelligence.

Who did you talk to, mate?

yup the most you can get from a hosting company is that months bill refunded.

there are companies that sell insurance for situations like this banks and big sites do buy insurance.

lol, that was funny

Clearly I thought fully managed hosting with RAID's and an extra $100 a month to have the site backed up daily on-site was enough. Nobody expects the entire 1000-server wide network to be destroyed by hackers.

We also had an offsite backup locally but its only done monthly and the data is useless.

Hindsight is always 20/20

rcn should release small bits of information they have on the hacker,

im sure someone on gfy would find him...

Wow. Sorry to hear it.

Probably a good time to spam.

Makes sense.

Number 1 . . . or are you talking about a different name?

:1orglaugh:thumbsup

shows 3rd for me, after some place that makes weed growing cabinets... then I clicked on the link... HELLO 1999! :) Seriously though, even if their backup server was a 96Tb raided NAS with twin gigabit NIC's capable of feeding at 2000mbits/s, it would take about 100+ hours just to transfer the data of 1000 servers, assuming 100% throughput which is hardly the case with encrypted NAS data...

Not to mention that even after it is restored, there are configuration problems and setup issues that need to be attended to as the servers are not backed up bit for bit... Doesn't take knowledge of a tape drive to figure out why this is going to take a while. 4000 man hours is a very conservative estimate, not considering ongoing support to fix small configuration issues.

Do some math: 4000 man hours at tech costs + overtime + contracts, say avg $25/hr and that is conservative... $100k plus a month worth of refunds (1000 servers at ~450$ a piece, $450,000) and lost business (say a very conservative 20%) makes this a pretty impossible situation financially but despite that, it is being done. From what I understand backups are actually being restored and support is being offered so maybe just lighten up and let the bits transfer at whatever speed they may.

I agree that with a proper disaster plan this could have been much smoother but either way you look at it... I can't think of another host that would come back from having every single one of their servers corrupted at once.

4chan anons are pretty pissed that their favorite fap sites are offline

you should try getting them on the case

The point of a disaster recovery plan is to be able to recover from disaster in a short period of time.

A properly set up data centre should be able to replicate it's operations in the event of a 100% failure in a reasonable length of time.

RCN was charging top dollar for what eventually turned out to be zero effective disaster recovery contingencies and now the customers are paying.

However people who are running a $60k a month operation should probably have their own contingencies in place as well.

can anyone post their new updates, how fast were you treated with isprime support?

this is why companies like sungard who specialize in services like charge what they charge.

A simple 500/month machine with someone like mojo, natnet or us would easily run you 2k/month+ under sungard.

If you look at your average adult focused hosting company and then look at a mainstream focused management company. The two extremes are different than what you would think. A simple dual quad with raid and 100Mbps in adult will run you 400-700/month. Rackspace/another higher end mainstream focused management company will charge 400-700 just for the box and in many cases 1k+, and then your looking at easily 10-40Mbps on banwdith on top of it.

What a lot of users fail to realize is the true cost behind the scenes that is involved in running a managed services company. People cost money, equipment and software costs money and building/space costs money.


As RCN mentioned the backup system they have in place, much like many other providers have in place are probably built around 50-100 machines per back up node, depending on hardware involved, and even then with 100 machines hitting that box each night you better schedule them carefully. These systems are not designed to mass restore dozens of machines at once. No way no how.

If someone ever wants to see how DR is done and the real costs involved. Just go lookup on google DR services and ask for a quote and see whats involved.

order a shit

ISprime was less than enthusiastic about my call to them setting up my box, the guy there sounded just as jaded and hectic as the RCN techs except this guy was in the sales dept. which really freaks me out. Could tell by the way this guy talked to me it's going to be a long time before I get my other box restored. :disgust

Why din't they take Mojohost Brad's offer to fly in some techs at his own expense to help restore the boxses. RNC did say they needed all the manpower they could get.

So here's Brads chance to prove weather he is the man of his word.

Yup, put it this way, they were colocating/leasing boxes from isprime who charges 660/month for a dual quad core/8gb ram+dual 15k raid and 10mbit commit... rcn wasn't charging much more than that maybe even the same or less, plus you had dedicated support who actually knew what they were doing and backup support that was great compared to the competition and without explicitly asking for it. I think a lot of people are overlooking 1 fact: RCN is still working their asses off to fix this problem despite initially facing some pretty grim circumstances. I checked in at 7am (it happened at 6:08am for the most part) and they already had the senior techs on it. How fucked would you be if your server went down AND your personal computer AND your work computer.

Honestly, I think this is the first time in several years that I've seen downtime with an RCN server. You can bet your ass that what doesn't break you will make you stronger... I still have a lot of faith in RCN despite the recent events. ISPRIME are really a pain to deal with already, "send an email here, contact so and so, contact us back after, no eta", typical left hand not talking to the right hand. RCN you hop on icq, ask them to set up lighttpd and a new domain on a new ip and you get a message back 5 minutes later when it's done.

I cant help but agree, the guys at ISPrime just answer with "Hello?" or "Yes?" .. Not how you would expect a professional to answer the phone at all.

shut up fuck face.

if you are who i think you are, it's better that your sites are offline.

warez anyone?

I wasn't too thrilled with my call to them either. Seems too hectic and I don't like not having an answer. I was just told to send them an email with my server details and they will respond. It's been about 8 hours with no response :(

I would prefer mojohost to step in as well.

yeah, I forgot that part too me a sec to realize I wasn't patched through to some kids house. I don't get it.:Oh crap

Couldn't agree more. They definitely had a very intimate and cool way of doing business and I respected them for that. I loved being able to drop them a message on ICQ to set up a new mail account or whatever and it was done within minutes. It was almost like having your own 24/7 admin on your payroll without the cost and a lot more reliable. Then this happened. I think they should reinvent themselves on a new network (not ISPrime.. maybe webair?) where they get a better wholesale rate so they can offer a more affordable price and make a bigger margin.

isprime never needed customer service

yeah when i called i had the same experience