![]() |
question about robots.txt
I have a static website and no robots.txt file
I login into awstats and see: Googlebot 19 302.54 KB 08 Dec 2012 - 07:37 I login into webmaster tools and see: Over the last 24 hours, Googlebot encountered 4 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%. So what is the problem here, if google crawls my site without robots.txt why do I need to add it? Especially when I do not want to disallow anything, so my robots.txt would look: Quote:
|
I always add an empty one just so i don't have to see the error messages :-)
|
Quote:
|
Quote:
Conclusion that robots.txt should be only used when you need to disallow something? And if you do not want to disallow something there is ZERO point of using it (except that error message)? |
Every site should have a robots.txt file. :2 cents:
|
Quote:
The whole point of robots.txt is to exclude bots. If you don't want to exclude anything, then you don't need one. I always add one anyways, though. |
Quote:
If its empty and spiders crawl your site without it - whats the point? |
there is no point, but add it anyway to be safe...
|
on sites i dont need to exclude i dont use them
|
Quote:
"Over the last 24 hours, Googlebot encountered 4 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%." Why did I get it now? There was no robots.txt file for months yet I got this message today... |
Is it really that hard to make a blank robots.txt file?
|
Quote:
7 and 10 posts are my questions that i am particularly interested in. |
Quote:
|
Quote:
Why does that message say that google did not crawl the site today after 4 attempts while they crawled it ok for the past 2 months? Why absence of robots.txt stopped them only today and not every day before to day? |
Quote:
|
Quote:
then press 7 for robots.txt support... :2 cents: |
Quote:
|
Quote:
|
Quote:
|
Quote:
|
Quote:
2. Then do not reply if you do not want to answer. Simple? |
Quote:
|
All times are GMT -7. The time now is 01:00 PM. |
Powered by vBulletin® Version 3.8.8
Copyright ©2000 - 2025, vBulletin Solutions, Inc.
©2000-, AI Media Network Inc123