View Single Post
Old 07-05-2007, 10:48 AM  
PinPine
Confirmed User
 
Join Date: Mar 2007
Posts: 385
Robots.txt question

Guys,

Quick question about robots.txt
If this following is my robots.txt I can assume that google wont index /feed /trackback etc right ?

I have placed robots.txt in my root directory and for some reason he is still indexing these pages e.g. www.teensqueezer.com/feed/ can be found on google. HOW ?

User-agent: Googlebot
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /rss/
Disallow: /comments/feed/
Disallow: /page/
Disallow: /date/
Disallow: /comments/
Disallow: /linkex/
__________________
ICQ: 490974172
Looking for hardlink trades: www.solidteens.com/linkex/
PinPine is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook