I have a static website and no robots.txt file
I login into awstats and see:
Googlebot 19 302.54 KB 08 Dec 2012 - 07:37
I login into webmaster tools and see:
Over the last 24 hours, Googlebot encountered 4 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%.
So what is the problem here, if google crawls my site without robots.txt why do I need to add it? Especially when I do not want to disallow anything, so my robots.txt would look:
What is the point of adding empty robots.txt if google crawls site anyway (according to my awstats)?