Thread: Robots.txt
View Single Post
Old 10-28-2008, 02:10 AM  
BigPimpCash
Confirmed User
 
Join Date: Jul 2006
Posts: 808
Question

Quote:
Originally Posted by d-null View Post
I add this to all of my sites' robots.txt :


User-agent: ia_archiver
Disallow: /




that keeps archive.org from spidering and keeping a copy of your site forever
Can I ask why this is a bad thing ? Isnt ia_archiver the alexa one ?
__________________

BIG PIMP CASH - Upto 70% Revshare and Upto 30$ PPS
[email protected] - ICQ - 197119155

Need Content ? TRASHY CONTENT
BigPimpCash is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote