Quote:
Originally Posted by mineistaken
1. You gave me link where is no explanation why 3 months google crawled my site and suddenly they decided not to crawl because of no robots. Maybe it was international robots day 
2. Then do not reply if you do not want to answer. Simple? |
Short answer: Google is now requiring robots.txt to return 200 or they give you warnings in Google Webmasters (Where any serious webmaster should already be anyway) to fix it or they stop crawling you.