havent heard that name in awhile

back of the line.

Damn, looks like that rogue employee really fucked shit up, huh?

Hope all other hosts learn from this and review their security.

I'll buy an ISprime server if it gets me my data back, but it doesn't sound like they have it.

RCN you are a piece of shit. You string me along for 3 fucking days tehn i find out you were never backing up my shit to begin with. I want you out of business.

How did you find out they weren't keeping your backups? Who did you talk to? And what kind of server were you packing?

i jsut called IS prime or wahtever its called and they said rcn was never backing up my databases and they will attempt to do a restore and to call them back in 3 hours. I basically lsot about $60,000 over this.

Wondering which sponsors are effected?

They are the little red dots in your stats remote.

Holy shit. :Oh crap

That's fucked :(

Did you have a backup service from RCN with your hosting deal? (it was an extra paid service)

isprime is good people, top notch with support and reliability.

Nightmare starts now....

"client 1: we have a backup of your content"
"client 2: we don't have a backup of your content"
"client 3: we have a backup of your content"
etc...

Posted on another forum from RCN today:

Lol, silent takeover! :P

so basically they are out of business?

that has already been posted Abbie, living under a rock?

yes I am ... i was being a lazy ho and not reading the entire thread before having coffee.

that's what i'd like to know

From what I understand unless you are buying a new server with ISPrime or staying with RCN, they are holding your data hostage. I've spoken to a lawyer and I'm considering starting a class action lawsuit due to financial losses involved and holding our data hostage after failing their duty of care.

Anyone interested in jumping on the bandwagon please email here:

[email protected]

Did epassporte own part of RCN ?

I hear the owners of RCN are releasing a huge blockbuster movie soon...

Don't all hosting companies have some fine print saying they can't be sued for downtime?

There is probably an accountant hiding all the money now for the time they actually pull the plug.

RCN are showing very bad form here. They should step up to the plate and be candid about what has really happened and why they can't restore peoples backups. There is more to this than 'it was taking too long' - I am sure of that.

i hope the guy who did this falls off a cliff face first into jagged rocks, lies there disabled in pain until the tide comes in to drown him.

Wow so if I read everything correctly here, RCN is not able to restore everything. That's a major fail!

One word: Clonebox

For those of you who may be unaware, we offer off-site backup services for overwhelmingly economical fees. I think this thread alone is reason enough to take a look: https://www.bettercgi.com/clonebox/

One word: Toolate

:1orglaugh:1orglaugh:1orglaugh


call dat dude from mojo host here's what he will say:

I will dink a red bull, sprout some wings, sprinkle some pixie dust on my techs,
fly out there to help, an we will simultaneously shit you your backups!

all fer free! :laughing-:laughing-:small-smi:small-smi

Here are the facts:

The network was attacked on Saturday morning at 5:30AM. The attackers were able to break into the management servers, from there they were able to get to all the customer servers that we manage.

They ran malicious commands to overwrite the filesystems corrupting the servers to a point where they need to be reformatted.

Our backup system was never meant to restore so many servers at once, it was designed to help an individual customer recover a single server, or some files in case of accidental deletion, or other action.

We manage nearly 1,000 servers and each server currently takes nearly 4 hours to recover, in other words 4000 man hours. This is not to mention that the backups are compressed, some customers have more data than others, and even when everything is restored there is still configuration work that is required to get everything working, ultimately this means that its over 4000 hours to get this completed.

We are simply not capable of recovering this many servers in any reasonable period of time, it will take over 1 month to get this done, and by then the customers will lose their businesses as well.

The backup system uses a custom format to compress and store the files so it is not easy to provide direct access to the data. In other words in order for anyone to get their files we need to go through a lengthly, and complicated restore procedure.

We wish nothing more than for this to be business as usual, but we are simply incapable of dealing with the circumstances.

Additionally the servers for our customers are on lease, these leases are owned by banks and from our projections the business would not be able to generate enough revenue to continue. That means that even if we brought up customers onto our network, in some time we would still go out of business due to loss of revenue and piling liabilities.

We have been working around the clock as we were not prepared for an event of this magnitude, the action that was carried out is malicious and criminal and we do not have a way to properly come back from this.

We are doing our best to work with each customer and get their servers back online, but we are simply overwhelmed. Some of our servers were hosted in ISPrime's facility, our other main facility is in North Bergen, NJ just a couple of miles away from ISPrime - and we are giving customers another option. We are not holding data hostage, and allow customers to move to any host that they choose. Please understand that we have nearly 200 customers and 1000 servers, and can only do single digit number of restores at a time, making it impossible to get to everyone's request at once.

Please bare with us as we look to restore your information and find a new home for your business.

We would also like to say that we do not concretely know who the attacker was, or what their intentions were, but we have opened a case with the FBI to investigate this matter and find out who is responsible for this malicious and criminal attack.

so you lied when you said it was an ex-employee?

"As for the intrusion it was the result of an ex-employee who was with us for three years as a result he had intimate knowledge of our systems which is why the effects are so large."

Wow......

Probably the biggest hack in history!

Has something like this ever happened before?

RCN have finally copied my sql dumps to my new server. Thankyou RCN. Now where are those red bulls..

Downtime is one thing - paying for a backup service you can't access to right these wrongs is totally another...

the official email says all backup data has been transferred over to isprime - so the onus is with them....

please get me access to this backup data so I can move it to another box to get sites back online - this is what a backup service was paid for... to put right a wrong quickly and efficiently.

I hope ISPrime can show their mettle.

If you run a 1000 server facility then shouldn't you also have a backup system to cope with , well, the destruction of the facility ?

I called ISPrime 5 times about this, their stance is they are helping out with customers migrating to ISPrime only. They have access to the backups for this purpose. If you are migrating to any other network you need to contact RCN. God speed to them.

I really hope the person responsible for this attack is brought to justice.

What a horrible situation!

If anybody needs help in getting their hardware back, I am located in New Jersey and can mail them out to you.

Yeah I noticed that as well - the story has gone from ex employee to hackers, with an s at the end. Are you sure you're not just making this up as you go along?

Perhaps you might want to post this information on your website as well. I'm sure you must have customers that don't get there news from adult message boards. And your website hasn't changed its message since it came back up.

http://www.realitychecknetwork.com/

Based on our experience, the "it would take over a month to restore everything" line is not
is strange as it sounds. A lot of people still use magnetic TAPE to back up. As a 1954 technology,
backup tapes can restore about 1 MB / second, or 86 GB / day. Only 2,600 GB would take a
month.
That's best case perfromance assuming the tech is sitting there the whole time watching closely
and immediately switching tapes and such. A mistake requiring a "do over" doubles the time,
of course.
So it's not unusual for restores to take a long time. I completely believe that, since most backup
systems I've seen have huge problems, and being super slow is a very common problem.

Personally, I'm not fond of living in 1954, so for Clonebox we use 16 drive wide RAID arrays
capable of pushing about 100 times that much throughput. If it would take a month to restore
all of the servers from tape, Clonebox could do it in about eight hours, or the sites could simply
run from the Clonebox servers, which takes only a few minutes to set up.

Acts of god aren't covered by your insurance either, this being the equivalent of that. Be grateful the backups weren't corrupted.

True that. However if you run a large facility then having a tape only backup system that can only cope with a few restores at a time is tantamount to not having a backup because the point of a backup is that you can deploy a restore quickly if need be.

Disaster recovery plans should cater for the worst case scenarios and obviously RCN, for all the hype of their managed server operations, didn't put enough work into developing an effective disaster recovery plan.

Now it seems they are pulling the plug on all these servers and forgoing the leases on them while giving ISPrime access to the backups. If what they say is true then regardless of who does the work, 4000 man hours of work is required to complete a restore. That is disastrous.

If you are running mission critical applications then you should insist that your host provides rotating offsite disk based backups so that disaster recovery is possible, because no disaster recovery plan is successful if it takes a month. :2 cents:

Note to self : Hackers have finally attained the status of God.

:1orglaugh

If a couple of days of downtime loses you $60k why don't you have a duplicated server (or servers) at another host?

Given the low cost of servers these days, you'd think this would be obvious wouldn't you ?

This is a good time for everyone to reflect on what would happen if their host went up in a puff of smoke and take steps to ensure they have some redundancy in their operation at all mission critical levels.

its not lost sales, its a lost database with 60k in rebills. And yes i do have backups of everythign 10 times over but not the userbase which they told me was being backed up with double redundancy.

Regardless, i finally made the proper threats and talked to the right people and got my files. so I am fine now. they were making up the proper backups all along, i just never got the right person on the horn.

Comment still stands: you should be making an offsite backup of that very valuable database at the least, even to the HD on your home computer.

Glad you got your data back, now consider taking better precautions going forward. :thumbsup

Personally, I wouldn't trust my business to anyone who's own website doesn't come up when you google their entire name.

Being a vulture wont likely net you any clients and implying that any serious hosts still uses tape storage, namely isprime, is an insult to your intelligence.

Who did you talk to, mate?

yup the most you can get from a hosting company is that months bill refunded.

there are companies that sell insurance for situations like this banks and big sites do buy insurance.