Quote:
Originally Posted by d-null
I add this to all of my sites' robots.txt :
User-agent: ia_archiver
Disallow: /
that keeps archive.org from spidering and keeping a copy of your site forever
|
Can I ask why this is a bad thing ? Isnt ia_archiver the alexa one ?