GoFuckYourself.com - Adult Webmaster Forum - Does anyone even bother with robots.txt anymore?

GoFuckYourself.com - Adult Webmaster Forum (https://gfy.com/index.php)

- Fucking Around & Business Discussion (https://gfy.com/forumdisplay.php?f=26)

- - Does anyone even bother with robots.txt anymore? (https://gfy.com/showthread.php?t=1097433)

xxxjay

01-24-2013 01:48 AM

Does anyone even bother with robots.txt anymore?

??????????

Mr Pheer

01-24-2013 01:48 AM

Of course, it's important.

EddyTheDog

01-24-2013 01:49 AM

Google says you should - I don't argue with Google....

xxxjay

01-24-2013 01:51 AM

try some results with it and try some without it, then get back to me.

EddyTheDog

01-24-2013 01:59 AM

Quote:

Originally Posted by xxxjay (Post 19441358)

try some results with it and try some without it, then get back to me.

You asked the question - You do the test and get back to me - Jebus.....

:upsidedow

bean-aid

01-24-2013 02:34 AM

Quote:

Originally Posted by xxxjay (Post 19441358)

try some results with it and try some without it, then get back to me.

What really pisses me off is when I try to google a white label and it says...

"robots.txt is preventing"

That would be enough for me...

Mr Pheer

01-24-2013 03:23 AM

Quote:

Originally Posted by xxxjay (Post 19441358)

try some results with it and try some without it, then get back to me.

Yeah I'll get right on that. Hold your breath, it wont take me long.

ottopottomouse

01-24-2013 06:29 AM

Always .

Rockhound

01-24-2013 07:26 AM

I use it because Google wants me to, but while we're on the subject... What do you guys put in your robots.txt files that make them better?

Barry-xlovecam

01-24-2013 07:50 AM

If you don't want a robots.txt 404 messing up your access logs -- make a blank file with that name -- most legit robots will request it and obey it (sometimes?).

The less 404s you have the better for many reasons ... I use a favicon.ico or a blank favicon.ico to avoid all the 404 errors too.

slapass

01-24-2013 07:54 AM

Quote:

Originally Posted by Barry-xlovecam (Post 19441655)

If you don't want a robots.txt 404 messing up your access logs -- make a blank file with that name -- most legit robots will request it and obey it (sometimes?).

The less 404s you have the better for many reasons ... I use a favicon.ico or a blank favicon.ico to avoid all the 404 errors too.

agreed.
:thumbsup

Porndex

01-24-2013 07:58 AM

I use it to block stuff that wouldn't be worth indexing. Works a treat with the major engines.

All times are GMT -7. The time now is 02:00 AM.