GoFuckYourself.com - Adult Webmaster Forum

GoFuckYourself.com - Adult Webmaster Forum (https://gfy.com/index.php)
-   Fucking Around & Business Discussion (https://gfy.com/forumdisplay.php?f=26)
-   -   question about robots.txt (https://gfy.com/showthread.php?t=1092239)

mineistaken 12-09-2012 02:39 PM

question about robots.txt
 
I have a static website and no robots.txt file

I login into awstats and see:
Googlebot 19 302.54 KB 08 Dec 2012 - 07:37

I login into webmaster tools and see:
Over the last 24 hours, Googlebot encountered 4 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%.

So what is the problem here, if google crawls my site without robots.txt why do I need to add it? Especially when I do not want to disallow anything, so my robots.txt would look:

Quote:

User-agent: *
What is the point of adding empty robots.txt if google crawls site anyway (according to my awstats)?

CYF 12-09-2012 02:57 PM

I always add an empty one just so i don't have to see the error messages :-)

ottopottomouse 12-09-2012 03:17 PM

Quote:

Originally Posted by CYF (Post 19360082)
I always add an empty one just so i don't have to see the error messages :-)

I do the same.

mineistaken 12-09-2012 03:19 PM

Quote:

Originally Posted by CYF (Post 19360082)
I always add an empty one just so i don't have to see the error messages :-)

So you are saying that robots.txt is not helping anything (except that you stop seeing that error message)?

Conclusion that robots.txt should be only used when you need to disallow something? And if you do not want to disallow something there is ZERO point of using it (except that error message)?

Killswitch 12-09-2012 04:34 PM

Every site should have a robots.txt file. :2 cents:

CYF 12-09-2012 04:37 PM

Quote:

Originally Posted by mineistaken (Post 19360109)
Conclusion that robots.txt should be only used when you need to disallow something? And if you do not want to disallow something there is ZERO point of using it (except that error message)?


The whole point of robots.txt is to exclude bots. If you don't want to exclude anything, then you don't need one. I always add one anyways, though.

mineistaken 12-09-2012 04:38 PM

Quote:

Originally Posted by Killswitch (Post 19360195)
Every site should have a robots.txt file. :2 cents:

My question is why? Except when you want to disallow something.
If its empty and spiders crawl your site without it - whats the point?

woj 12-09-2012 04:55 PM

there is no point, but add it anyway to be safe...

Fat Panda 12-09-2012 05:02 PM

on sites i dont need to exclude i dont use them

mineistaken 12-09-2012 05:07 PM

Quote:

Originally Posted by SAC (Post 19360232)
on sites i dont need to exclude i dont use them

Same here... But now I got email from webmaster tools:

"Over the last 24 hours, Googlebot encountered 4 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%."

Why did I get it now? There was no robots.txt file for months yet I got this message today...

Killswitch 12-09-2012 05:13 PM

Is it really that hard to make a blank robots.txt file?

mineistaken 12-09-2012 05:54 PM

Quote:

Originally Posted by Killswitch (Post 19360241)
Is it really that hard to make a blank robots.txt file?

It looks like you are completely missing the point of this thread. I do not want just to stop that webmaster tools error, I want to understand how it works.
7 and 10 posts are my questions that i am particularly interested in.

Killswitch 12-09-2012 06:01 PM

Quote:

Originally Posted by mineistaken (Post 19360297)
It looks like you are completely missing the point of this thread. I do not want just to stop that webmaster tools error, I want to understand how it works.
7 and 10 posts are my questions that i am particularly interested in.

You know what those errors mean in Google Webmasters? 404... They couldn't find it... Make a blank one, errors stop.

mineistaken 12-09-2012 08:19 PM

Quote:

Originally Posted by Killswitch (Post 19360306)
You know what those errors mean in Google Webmasters? 404... They couldn't find it... Make a blank one, errors stop.

Why did I get that error for the first time only today while I did not have robots.txt all the time (about 2-3 months with that site added to webmaster tools account)

Why does that message say that google did not crawl the site today after 4 attempts while they crawled it ok for the past 2 months? Why absence of robots.txt stopped them only today and not every day before to day?

Killswitch 12-10-2012 07:00 AM

Quote:

Originally Posted by mineistaken (Post 19360467)
Why did I get that error for the first time only today while I did not have robots.txt all the time (about 2-3 months with that site added to webmaster tools account)

Why does that message say that google did not crawl the site today after 4 attempts while they crawled it ok for the past 2 months? Why absence of robots.txt stopped them only today and not every day before to day?

http://support.google.com/webmasters/?hl=en

woj 12-10-2012 07:22 AM

Quote:

Originally Posted by Killswitch (Post 19360940)

calling them is 10x more effective, 1-800-GOOGLE
then press 7 for robots.txt support... :2 cents:

BIGTYMER 12-10-2012 07:37 AM

Quote:

Originally Posted by woj (Post 19360968)
calling them is 10x more effective, 1-800-GOOGLE
then press 7 for robots.txt support... :2 cents:

:1orglaugh

Killswitch 12-10-2012 07:47 AM

Quote:

Originally Posted by woj (Post 19360968)
calling them is 10x more effective, 1-800-GOOGLE
then press 7 for robots.txt support... :2 cents:

Not sure if you're joking or have seriously called Google about robots.txt support. :1orglaugh

mineistaken 12-10-2012 08:58 AM

Quote:

Originally Posted by Killswitch (Post 19360940)

Ok, you have no answer to my questions, yet you still continue posting. Stop please :)

Killswitch 12-10-2012 09:02 AM

Quote:

Originally Posted by mineistaken (Post 19361151)
Ok, you have no answer to my questions, yet you still continue posting. Stop please :)

No, I'm just not your personal fucking Google, learn to search yourself.

mineistaken 12-10-2012 03:45 PM

Quote:

Originally Posted by Killswitch (Post 19361163)
No, I'm just not your personal fucking Google, learn to search yourself.

1. You gave me link where is no explanation why 3 months google crawled my site and suddenly they decided not to crawl because of no robots. Maybe it was international robots day :1orglaugh

2. Then do not reply if you do not want to answer. Simple?

Killswitch 12-10-2012 04:22 PM

Quote:

Originally Posted by mineistaken (Post 19362253)
1. You gave me link where is no explanation why 3 months google crawled my site and suddenly they decided not to crawl because of no robots. Maybe it was international robots day :1orglaugh

2. Then do not reply if you do not want to answer. Simple?

Short answer: Google is now requiring robots.txt to return 200 or they give you warnings in Google Webmasters (Where any serious webmaster should already be anyway) to fix it or they stop crawling you.


All times are GMT -7. The time now is 01:00 PM.

Powered by vBulletin® Version 3.8.8
Copyright ©2000 - 2025, vBulletin Solutions, Inc.
©2000-, AI Media Network Inc123