question about robots.txt
 

I have a static website and no robots.txt file

I login into awstats and see:
Googlebot 19 302.54 KB 08 Dec 2012 - 07:37

I login into webmaster tools and see:
Over the last 24 hours, Googlebot encountered 4 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%.

So what is the problem here, if google crawls my site without robots.txt why do I need to add it? Especially when I do not want to disallow anything, so my robots.txt would look:

What is the point of adding empty robots.txt if google crawls site anyway (according to my awstats)?

I always add an empty one just so i don't have to see the error messages :-)

I do the same.

So you are saying that robots.txt is not helping anything (except that you stop seeing that error message)?

Conclusion that robots.txt should be only used when you need to disallow something? And if you do not want to disallow something there is ZERO point of using it (except that error message)?

Every site should have a robots.txt file. :2 cents:

The whole point of robots.txt is to exclude bots. If you don't want to exclude anything, then you don't need one. I always add one anyways, though.

My question is why? Except when you want to disallow something.
If its empty and spiders crawl your site without it - whats the point?

there is no point, but add it anyway to be safe...

on sites i dont need to exclude i dont use them

Same here... But now I got email from webmaster tools:

"Over the last 24 hours, Googlebot encountered 4 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%."

Why did I get it now? There was no robots.txt file for months yet I got this message today...

Is it really that hard to make a blank robots.txt file?

It looks like you are completely missing the point of this thread. I do not want just to stop that webmaster tools error, I want to understand how it works.
7 and 10 posts are my questions that i am particularly interested in.

You know what those errors mean in Google Webmasters? 404... They couldn't find it... Make a blank one, errors stop.

Why did I get that error for the first time only today while I did not have robots.txt all the time (about 2-3 months with that site added to webmaster tools account)

Why does that message say that google did not crawl the site today after 4 attempts while they crawled it ok for the past 2 months? Why absence of robots.txt stopped them only today and not every day before to day?

calling them is 10x more effective, 1-800-GOOGLE
then press 7 for robots.txt support... :2 cents:

:1orglaugh

Not sure if you're joking or have seriously called Google about robots.txt support. :1orglaugh

Ok, you have no answer to my questions, yet you still continue posting. Stop please :)

No, I'm just not your personal fucking Google, learn to search yourself.

1. You gave me link where is no explanation why 3 months google crawled my site and suddenly they decided not to crawl because of no robots. Maybe it was international robots day :1orglaugh

2. Then do not reply if you do not want to answer. Simple?

Short answer: Google is now requiring robots.txt to return 200 or they give you warnings in Google Webmasters (Where any serious webmaster should already be anyway) to fix it or they stop crawling you.