Quote:
Originally Posted by thommy
a weapon company uploads the newest secret version of a killer machine into their web - Google crawls it and publish it without the explicit demand of doing so - they would be also in trouble.
|
Every day thousands of others bots scan your website, ahref, majestic, exploit looking bots, advert bots, other shit bots, most of them have loaded default directories and file names or directory paths for scripts working on your site. If you do not want something to appear on the Internet, you do not upload to the internet. Simple.[/QUOTE]
Quote:
Originally Posted by thommy
I think that robots.txt would be the simplest way to allow or deny to crawl and publish
stuff from a site.
|
If in the robot file you select which file or directory to bypass the possible that Google will do. But for others it will be a gift.