Thread: Does anyone even bother with robots.txt anymore?
View Single Post
Old 01-24-2013, 02:34 AM  
bean-aid
So Fucking Banned
 
Industry Role:
Join Date: Jun 2011
Location: the land of woke sleuths
Posts: 16,493
Quote:
Originally Posted by xxxjay View Post
try some results with it and try some without it, then get back to me.
What really pisses me off is when I try to google a white label and it says...

"robots.txt is preventing"

That would be enough for me...
bean-aid is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote