Thread: Robots.txt
View Single Post
Old 10-28-2008, 08:20 AM  
NinjaSteve
Too lazy to set a custom title
 
Industry Role:
Join Date: Dec 2003
Posts: 11,089
Quote:
Originally Posted by d-null View Post
I add this to all of my sites' robots.txt :


User-agent: ia_archiver
Disallow: /




that keeps archive.org from spidering and keeping a copy of your site forever
Seems to be more than just the archive.org, but alexa overall.
http://www.alexa.com/site/help/webmasters
__________________
...
Last edited by NinjaSteve; 10-28-2008 at 08:23 AM..
NinjaSteve is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote