View Single Post
Old 01-03-2009, 05:42 PM  
rowan
Too lazy to set a custom title
 
Join Date: Mar 2002
Location: Australia
Posts: 17,393
I was going to suggest they could probably restore a good chunk of entries via archive.org (even if they just turned it into a historical archive), but they're following the "accept some, deny all" robots.txt model which means that archive.org isn't allowed to crawl the site. If that block has been in place for some time then there's no chance of any recent posts being in the archive.

User-agent: *
Disallow: /
User-agent: kinja-imagebot
Disallow:
User-agent: kinjabot
Disallow:
User-agent: Mediapartners-Google*
Disallow:
User-agent: Googlebot
Disallow:
User-agent: msnbot
Disallow:


Hmm... Google, Google Adsense, MSN/Live, and kinja are the only bots permitted to crawl the site.

Where's Yahoo? Blocked......
rowan is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote