GoFuckYourself.com - Adult Webmaster Forum - View Single Post - Something you may not know about robots.txt

fris · 07-27-2011, 05:58 PM

3 Things about robots.txt I learned today.

you can ignore a certain file type

Code:

Disallow: /*.avi$

you can ignore a query string

Code:

Disallow: /*customerID=

hide directories without revealing the name of the directory (robots.txt ignores
directory names starting with first two characters)

Code:

Disallow: /as
Disallow: /ca
Disallow: /cg
Disallow: /do
Disallow: /la
Disallow: /ma
Disallow: /op
Disallow: /st

07-27-2011, 05:58 PM
fris Too lazy to set a custom title Industry Role: Join Date: Aug 2002 Posts: 55,248	Something you may not know about robots.txt 3 Things about robots.txt I learned today. you can ignore a certain file type Code: Disallow: /.avi$ you can ignore a query string Code: Disallow: /customerID= hide directories without revealing the name of the directory (robots.txt ignores directory names starting with first two characters) Code: Disallow: /as Disallow: /ca Disallow: /cg Disallow: /do Disallow: /la Disallow: /ma Disallow: /op Disallow: /st __________________ Since 1999: 69 Adult Industry awards for Best Hosting Company and professional excellence. WP Stuff