|
|
|
|
||||
|
Welcome to the GoFuckYourself.com - Adult Webmaster Forum forums. You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features. Registration is fast, simple and absolutely free so please, join our community today! If you have any problems with the registration process or your account login, please contact us. |
|
|||||||
| New Webmasters ask "How-To" questions here. This is where other fucking Webmasters help. |
|
|
Thread Tools |
07-05-2007, 10:48 AM
|
#1 |
|
Confirmed User
Join Date: Mar 2007
Posts: 385
|
Robots.txt question
Guys,
Quick question about robots.txt If this following is my robots.txt I can assume that google wont index /feed /trackback etc right ? I have placed robots.txt in my root directory and for some reason he is still indexing these pages e.g. www.teensqueezer.com/feed/ can be found on google. HOW ? User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ User-agent: * Disallow: /wp- Disallow: /feed/ Disallow: /trackback/ Disallow: /rss/ Disallow: /comments/feed/ Disallow: /page/ Disallow: /date/ Disallow: /comments/ Disallow: /linkex/ |
|
|
|
07-05-2007, 04:19 PM
|
#2 |
|
Amateur Pimpin
Industry Role:
Join Date: Aug 2004
Location: Orlando, FL
Posts: 13,075
|
you realize you may have to wait a while before the spider reads this new txt file right?
__________________
Make easy money with Webcams |
|
|
|
|
07-06-2007, 05:52 AM
|
#3 |
|
Confirmed User
Join Date: Mar 2007
Posts: 385
|
I did not know that! i thought the very first time the spider would come along, it would read the list....maybe I've got to have some more patience.
thnx |
|
|
|
|
07-06-2007, 06:25 AM
|
#4 |
|
Amateur Pimpin
Industry Role:
Join Date: Aug 2004
Location: Orlando, FL
Posts: 13,075
|
Yeah give it a little while, remember anytime you're dealing with SE or spiders, it's about patience, might take a while, esp if the site is newer.
__________________
Make easy money with Webcams |
|
|
|
|
07-26-2007, 03:46 AM
|
#5 |
|
Confirmed User
Join Date: Jun 2007
Posts: 2,005
|
Interesting!
__________________
 ----- Blog Themes for $49 ----- ------ INSTANT DELIVERY ------ |
|
|
|
