Thread: question about robots.txt
View Single Post
Old 12-09-2012, 04:38 PM  
mineistaken
See signature :)
 
mineistaken's Avatar
 
Industry Role:
Join Date: Apr 2007
Location: ICQ 363 097 773
Posts: 29,656
Quote:
Originally Posted by Killswitch View Post
Every site should have a robots.txt file.
My question is why? Except when you want to disallow something.
If its empty and spiders crawl your site without it - whats the point?
mineistaken is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote