question about robots.txt - GoFuckYourself.com

mineistaken · 12-09-2012, 02:39 PM

I have a static website and no robots.txt file

I login into awstats and see:
Googlebot 19 302.54 KB 08 Dec 2012 - 07:37

I login into webmaster tools and see:
Over the last 24 hours, Googlebot encountered 4 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%.

So what is the problem here, if google crawls my site without robots.txt why do I need to add it? Especially when I do not want to disallow anything, so my robots.txt would look:

Quote:

User-agent: *

What is the point of adding empty robots.txt if google crawls site anyway (according to my awstats)?

CYF · 12-09-2012, 02:57 PM

I always add an empty one just so i don't have to see the error messages :-)

ottopottomouse · 12-09-2012, 03:17 PM

Quote:

Originally Posted by CYF

I always add an empty one just so i don't have to see the error messages :-)

I do the same.

mineistaken · 12-09-2012, 03:19 PM

Quote:

Originally Posted by CYF

I always add an empty one just so i don't have to see the error messages :-)

So you are saying that robots.txt is not helping anything (except that you stop seeing that error message)?

Conclusion that robots.txt should be only used when you need to disallow something? And if you do not want to disallow something there is ZERO point of using it (except that error message)?

Killswitch · 12-09-2012, 04:34 PM

Every site should have a robots.txt file.

CYF · 12-09-2012, 04:37 PM

Quote:

Originally Posted by mineistaken

Conclusion that robots.txt should be only used when you need to disallow something? And if you do not want to disallow something there is ZERO point of using it (except that error message)?

The whole point of robots.txt is to exclude bots. If you don't want to exclude anything, then you don't need one. I always add one anyways, though.

mineistaken · 12-09-2012, 04:38 PM

Quote:

Originally Posted by Killswitch

Every site should have a robots.txt file.

My question is why? Except when you want to disallow something.
If its empty and spiders crawl your site without it - whats the point?

woj · 12-09-2012, 04:55 PM

there is no point, but add it anyway to be safe...

Fat Panda · 12-09-2012, 05:02 PM

on sites i dont need to exclude i dont use them

mineistaken · 12-09-2012, 05:07 PM

Quote:

Originally Posted by SAC

on sites i dont need to exclude i dont use them

Same here... But now I got email from webmaster tools:

"Over the last 24 hours, Googlebot encountered 4 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%."

Why did I get it now? There was no robots.txt file for months yet I got this message today...

Killswitch · 12-09-2012, 05:13 PM

Is it really that hard to make a blank robots.txt file?

mineistaken · 12-09-2012, 05:54 PM

Quote:

Originally Posted by Killswitch

Is it really that hard to make a blank robots.txt file?

It looks like you are completely missing the point of this thread. I do not want just to stop that webmaster tools error, I want to understand how it works.
7 and 10 posts are my questions that i am particularly interested in.

Killswitch · 12-09-2012, 06:01 PM

Quote:

Originally Posted by mineistaken

It looks like you are completely missing the point of this thread. I do not want just to stop that webmaster tools error, I want to understand how it works.
7 and 10 posts are my questions that i am particularly interested in.

You know what those errors mean in Google Webmasters? 404... They couldn't find it... Make a blank one, errors stop.

mineistaken · 12-09-2012, 08:19 PM

Quote:

Originally Posted by Killswitch

You know what those errors mean in Google Webmasters? 404... They couldn't find it... Make a blank one, errors stop.

Why did I get that error for the first time only today while I did not have robots.txt all the time (about 2-3 months with that site added to webmaster tools account)

Why does that message say that google did not crawl the site today after 4 attempts while they crawled it ok for the past 2 months? Why absence of robots.txt stopped them only today and not every day before to day?

Killswitch · 12-10-2012, 07:00 AM

Quote:

Originally Posted by mineistaken

Why did I get that error for the first time only today while I did not have robots.txt all the time (about 2-3 months with that site added to webmaster tools account)

Why does that message say that google did not crawl the site today after 4 attempts while they crawled it ok for the past 2 months? Why absence of robots.txt stopped them only today and not every day before to day?

http://support.google.com/webmasters/?hl=en

woj · 12-10-2012, 07:22 AM

Quote:

Originally Posted by Killswitch

http://support.google.com/webmasters/?hl=en

calling them is 10x more effective, 1-800-GOOGLE
then press 7 for robots.txt support...

BIGTYMER · 12-10-2012, 07:37 AM

Quote:

Originally Posted by woj

calling them is 10x more effective, 1-800-GOOGLE
then press 7 for robots.txt support...

Killswitch · 12-10-2012, 07:47 AM

Quote:

Originally Posted by woj

calling them is 10x more effective, 1-800-GOOGLE
then press 7 for robots.txt support...

Not sure if you're joking or have seriously called Google about robots.txt support.

mineistaken · 12-10-2012, 08:58 AM

Quote:

Originally Posted by Killswitch

http://support.google.com/webmasters/?hl=en

Ok, you have no answer to my questions, yet you still continue posting. Stop please

Killswitch · 12-10-2012, 09:02 AM

Quote:

Originally Posted by mineistaken

Ok, you have no answer to my questions, yet you still continue posting. Stop please

No, I'm just not your personal fucking Google, learn to search yourself.

mineistaken · 12-10-2012, 03:45 PM

Quote:

Originally Posted by Killswitch

No, I'm just not your personal fucking Google, learn to search yourself.

1. You gave me link where is no explanation why 3 months google crawled my site and suddenly they decided not to crawl because of no robots. Maybe it was international robots day

2. Then do not reply if you do not want to answer. Simple?

Killswitch · 12-10-2012, 04:22 PM

Quote:

Originally Posted by mineistaken

1. You gave me link where is no explanation why 3 months google crawled my site and suddenly they decided not to crawl because of no robots. Maybe it was international robots day

2. Then do not reply if you do not want to answer. Simple?

Short answer: Google is now requiring robots.txt to return 200 or they give you warnings in Google Webmasters (Where any serious webmaster should already be anyway) to fix it or they stop crawling you.

12-09-2012, 02:57 PM	#2
CYF Coupon Guru Industry Role: Join Date: Mar 2009 Location: Minneapolis Posts: 10,973	I always add an empty one just so i don't have to see the error messages :-) __________________ Webmaster Coupons Coupons and discounts for hosting, domains, SSL Certs, and more! AmeriNOC Coupons \| Certified Hosting Coupons \| Hosting Coupons \| Domain Name Coupons

12-09-2012, 04:34 PM	#5
Killswitch 👏 REVOLUTIONARY 👏 Industry Role: Join Date: Oct 2012 Posts: 2,304	Every site should have a robots.txt file. __________________

12-09-2012, 04:55 PM	#8
woj <&(©¿©)&> Industry Role: Join Date: Jul 2002 Location: Chicago Posts: 47,882	there is no point, but add it anyway to be safe... __________________ Custom Software Development, email: woj#at#wojfun#.#com to discuss details or skype: wojl2000 or gchat: wojfun or telegram: wojl2000 Affiliate program tools: Hosted Galleries Manager Banner Manager Video Manager Wordpress Affiliate Plugin Pic/Movie of the Day Fansign Generator Zip Manager

12-09-2012, 05:13 PM	#11
Killswitch 👏 REVOLUTIONARY 👏 Industry Role: Join Date: Oct 2012 Posts: 2,304	Is it really that hard to make a blank robots.txt file? __________________

12-09-2012, 05:02 PM	#9
Fat Panda Porn is Dead. Move along. Industry Role: Join Date: Aug 2006 Posts: 13,295	on sites i dont need to exclude i dont use them