GoFuckYourself.com - Adult Webmaster Forum - Something you may not know about robots.txt

- Fucking Around & Business Discussion (https://gfy.com/forumdisplay.php?f=26)

- - Something you may not know about robots.txt (https://gfy.com/showthread.php?t=1031987)

Something you may not know about robots.txt

3 Things about robots.txt I learned today.

you can ignore a certain file type

Code:

Disallow: /*.avi$

you can ignore a query string

Code:

Disallow: /*customerID=

hide directories without revealing the name of the directory (robots.txt ignores
directory names starting with first two characters)

Code:

Disallow: /as

Disallow: /ca

Disallow: /cg

Disallow: /do

Disallow: /la

Disallow: /ma

Disallow: /op

Disallow: /st

Quote:

Originally Posted by fris (Post 18312667)

3 Things about robots.txt I learned today.

you can ignore a certain file type

Code:

Disallow: /*.avi$

you can ignore a query string

Code:

Disallow: /*customerID=

hide directories without revealing the name of the directory (robots.txt ignores
directory names starting with first two characters)

Code:

Disallow: /as

Disallow: /ca

Disallow: /cg

Disallow: /do

Disallow: /la

Disallow: /ma

Disallow: /op

Disallow: /st

Yes, you're meant to use it to disallow all the crap on your site.

But I didn't know about the last point. Also, it depends on the spider as to whether or not it listens to every factor in your robots.txt. Some spiders like googlebot are nice, and others are evil :mad:

I like the last one..

fris when you comin in for some traffic mang.. your acct is all set.