Something you may not know about robots.txt - GoFuckYourself.com

fris · 07-27-2011, 05:58 PM

3 Things about robots.txt I learned today.

you can ignore a certain file type

Code:

Disallow: /*.avi$

you can ignore a query string

Code:

Disallow: /*customerID=

hide directories without revealing the name of the directory (robots.txt ignores
directory names starting with first two characters)

Code:

Disallow: /as
Disallow: /ca
Disallow: /cg
Disallow: /do
Disallow: /la
Disallow: /ma
Disallow: /op
Disallow: /st

PornStarToys · 07-27-2011, 06:00 PM

Quote:

Originally Posted by fris

3 Things about robots.txt I learned today.

you can ignore a certain file type

Code:

Disallow: /*.avi$

you can ignore a query string

Code:

Disallow: /*customerID=

hide directories without revealing the name of the directory (robots.txt ignores
directory names starting with first two characters)

Code:

Disallow: /as
Disallow: /ca
Disallow: /cg
Disallow: /do
Disallow: /la
Disallow: /ma
Disallow: /op
Disallow: /st

Yes, you're meant to use it to disallow all the crap on your site.

But I didn't know about the last point. Also, it depends on the spider as to whether or not it listens to every factor in your robots.txt. Some spiders like googlebot are nice, and others are evil

Adam X · 07-27-2011, 11:42 PM

I like the last one..

fris when you comin in for some traffic mang.. your acct is all set.

07-27-2011, 05:58 PM	#1
fris Too lazy to set a custom title Industry Role: Join Date: Aug 2002 Posts: 55,247	Something you may not know about robots.txt 3 Things about robots.txt I learned today. you can ignore a certain file type Code: Disallow: /.avi$ you can ignore a query string Code: Disallow: /customerID= hide directories without revealing the name of the directory (robots.txt ignores directory names starting with first two characters) Code: Disallow: /as Disallow: /ca Disallow: /cg Disallow: /do Disallow: /la Disallow: /ma Disallow: /op Disallow: /st __________________ Since 1999: 69 Adult Industry awards for Best Hosting Company and professional excellence. WP Stuff

07-27-2011, 11:42 PM	#3
Adam X Now Giving 1 Fuck Daily Industry Role: Join Date: Apr 2002 Location: California Posts: 2,493	I like the last one.. fris when you comin in for some traffic mang.. your acct is all set. __________________ Cronfund - Buy CRON now and earn 4% plus coin value. The best of DEFI!