Quote:
Originally Posted by dcortez
I have watched googlebot test/spider for the existence of pages that neither existed on my websites, nor were ever linked to from anywhere. The spider would literally concoct URLS with random text and try them.
|
Yes I see it do that, I suspect it's to find out what your 404 page looks like. It has also in the past tried to find "/m/" and "/mobile/" pages that never existed, just in case you made a mobile version of your site and didn't think to link to it anywhere.