View Single Post
Old 12-09-2012, 04:37 PM  
CYF
Coupon Guru
 
CYF's Avatar
 
Industry Role:
Join Date: Mar 2009
Location: Minneapolis
Posts: 10,973
Quote:
Originally Posted by mineistaken View Post
Conclusion that robots.txt should be only used when you need to disallow something? And if you do not want to disallow something there is ZERO point of using it (except that error message)?

The whole point of robots.txt is to exclude bots. If you don't want to exclude anything, then you don't need one. I always add one anyways, though.
__________________
Webmaster Coupons Coupons and discounts for hosting, domains, SSL Certs, and more!
AmeriNOC Coupons | Certified Hosting Coupons | Hosting Coupons | Domain Name Coupons

CYF is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote