Quote:
Originally Posted by dcortez
You cannot actually "disallow" Googlebot from any path. It can, and does follow ALL links it finds, regardless of robots.txt settings.
My experience has shown this to be the case, for years.
|
The Googlebot that does traverse into blocked paths is checking for malware and malicious code rather than indexing. If you prefer, you can use .htaccess to further restrict Googlebot from entering into blocked paths (by useragent or IP).
WG