Does anyone even bother with robots.txt anymore? - GoFuckYourself.com

xxxjay · 01-24-2013, 01:48 AM

??????????

Mr Pheer · 01-24-2013, 01:48 AM

Of course, it's important.

EddyTheDog · 01-24-2013, 01:49 AM

Google says you should - I don't argue with Google....

xxxjay · 01-24-2013, 01:51 AM

try some results with it and try some without it, then get back to me.

EddyTheDog · 01-24-2013, 01:59 AM

Quote:

Originally Posted by xxxjay

try some results with it and try some without it, then get back to me.

You asked the question - You do the test and get back to me - Jebus.....

bean-aid · 01-24-2013, 02:34 AM

Quote:

Originally Posted by xxxjay

try some results with it and try some without it, then get back to me.

What really pisses me off is when I try to google a white label and it says...

"robots.txt is preventing"

That would be enough for me...

Mr Pheer · 01-24-2013, 03:23 AM

Quote:

Originally Posted by xxxjay

try some results with it and try some without it, then get back to me.

Yeah I'll get right on that. Hold your breath, it wont take me long.

ottopottomouse · 01-24-2013, 06:29 AM

Always .

Rockhound · 01-24-2013, 07:26 AM

I use it because Google wants me to, but while we're on the subject... What do you guys put in your robots.txt files that make them better?

Barry-xlovecam · 01-24-2013, 07:50 AM

If you don't want a robots.txt 404 messing up your access logs -- make a blank file with that name -- most legit robots will request it and obey it (sometimes?).

The less 404s you have the better for many reasons ... I use a favicon.ico or a blank favicon.ico to avoid all the 404 errors too.

slapass · 01-24-2013, 07:54 AM

Quote:

Originally Posted by Barry-xlovecam

If you don't want a robots.txt 404 messing up your access logs -- make a blank file with that name -- most legit robots will request it and obey it (sometimes?).

The less 404s you have the better for many reasons ... I use a favicon.ico or a blank favicon.ico to avoid all the 404 errors too.

agreed.

Porndex · 01-24-2013, 07:58 AM

I use it to block stuff that wouldn't be worth indexing. Works a treat with the major engines.

01-24-2013, 01:48 AM	#1
xxxjay Tube groupie. Industry Role: Join Date: Aug 2002 Location: LoScandalous, CA Posts: 13,483	Does anyone even bother with robots.txt anymore? ?????????? __________________ http://donttellmehowtoruinmylife.com/ - http://www.jmdigitalmarketing.com/my...s-and-reviews/ - http://www.wouldyouhitit.org - http://shinyobjectreviews.com/

01-24-2013, 01:49 AM	#3
EddyTheDog Just Doing My Own Thing Industry Role: Join Date: Jan 2011 Location: London, Spain, New Zealand, GFY - Not Croydon... Posts: 25,038	Google says you should - I don't argue with Google.... __________________ - Chaturbate Script - https://gfy.com/fucking-around-and-b...er-issues.html - Now supports White Labels

01-24-2013, 01:51 AM	#4
xxxjay Tube groupie. Industry Role: Join Date: Aug 2002 Location: LoScandalous, CA Posts: 13,483	try some results with it and try some without it, then get back to me. __________________ http://donttellmehowtoruinmylife.com/ - http://www.jmdigitalmarketing.com/my...s-and-reviews/ - http://www.wouldyouhitit.org - http://shinyobjectreviews.com/

01-24-2013, 06:29 AM	#8
ottopottomouse She is ugly, bad luck. Industry Role: Join Date: Jan 2010 Posts: 13,177	Always . __________________ ↑ see post ↑ 13101

01-24-2013, 07:26 AM	#9
Rockhound Confirmed User Industry Role: Join Date: Jan 2003 Posts: 334	I use it because Google wants me to, but while we're on the subject... What do you guys put in your robots.txt files that make them better? __________________ ALL BUSTY BABES - The Biggest Boobs On The Net

01-24-2013, 01:48 AM	#2
Mr Pheer Living inside your head. Industry Role: Join Date: Dec 2002 Location: In your AirBNB Posts: 20,488	Of course, it's important.

01-24-2013, 07:50 AM	#10
Barry-xlovecam It's 42 Industry Role: Join Date: Jun 2010 Location: Global Posts: 18,083	If you don't want a robots.txt 404 messing up your access logs -- make a blank file with that name -- most legit robots will request it and obey it (sometimes?). The less 404s you have the better for many reasons ... I use a favicon.ico or a blank favicon.ico to avoid all the 404 errors too.

01-24-2013, 07:58 AM	#12
Porndex Registered User Industry Role: Join Date: Oct 2012 Location: Germany Posts: 28	I use it to block stuff that wouldn't be worth indexing. Works a treat with the major engines.