GoFuckYourself.com - Adult Webmaster Forum - View Single Post - Google to make robots.txt an Internet standard after 25 years

rowan · 07-07-2019, 11:37 PM

Funny how Google is going on about making a de-facto a standard, when they explicitly ignore a fairly important (IMHO) de-facto directive: Crawl-delay.

Website: I'm asking you nicely to please limit your fetching to once per 60 seconds.

GoogleBot: No.

07-07-2019, 11:37 PM
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	Funny how Google is going on about making a de-facto a standard, when they explicitly ignore a fairly important (IMHO) de-facto directive: Crawl-delay. Website: I'm asking you nicely to please limit your fetching to once per 60 seconds. GoogleBot: No.