|
|
|
|
||||
|
Welcome to the GoFuckYourself.com - Adult Webmaster Forum forums. You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features. Registration is fast, simple and absolutely free so please, join our community today! If you have any problems with the registration process or your account login, please contact us. |
|
|||||||
| New Webmasters ask "How-To" questions here. This is where other fucking Webmasters help. |
|
|
Thread Tools |
07-30-2013, 11:25 AM
|
#1 |
|
Confirmed User
Industry Role:
Join Date: Nov 2005
Posts: 1,251
|
Simple question about robots.txt
Hi,
Maybe someone can help me with a little question that I have about robots.txt So doing a site:domain.com on google.com I found many search results that I don't want to appear. All links are: http://www.domain.com/foldername/filename.php I added this to my robots.txt, please let me know if is correct: Code:
User-agent: * Sitemap: http://www.domain.com/sitemap.xml Disallow: /link/ Disallow: /link/*.php$ Or one of this lines is enough? If yes, whats line is better? After doing this, is better to use webmaster tools to say google to remove that links from search results? I think that they have this option right? If is a good idea, can you please teach me by saying the steps that I need? Thank you guys! Best regards! |
|
|
|
07-30-2013, 07:45 PM
|
#2 |
|
Registered User
Industry Role:
Join Date: Nov 2012
Posts: 50
|
Disallow: /link/ [this will block all files within this directory]
Disallow: /link/*.php$ [this will block all .php files within this directory] Best is to try and test it within Webmaster tool. Cheers! |
|
|
|
|
07-31-2013, 02:28 AM
|
#3 |
|
Confirmed User
Industry Role:
Join Date: Nov 2005
Posts: 1,251
|
Thank you for your reply buddy.
But Webmaster Tools don't give any good help... he ignores the disallow lines. Screenshot: http://i.imgur.com/YDJmyI5.png |
|
|
|
|
07-31-2013, 04:41 AM
|
#4 |
|
Registered User
Industry Role:
Join Date: Nov 2012
Posts: 50
|
well, looks like you tried your root domain during testing. You need to try yourdomain dot com / links / yourphpfile.php to verify whether your robot.txt is working the way it's suppose to. You want to block .php files within link dir and not entire domain, correct?
|
|
|
|
|
07-31-2013, 08:53 AM
|
#5 |
|
Confirmed User
Industry Role:
Join Date: Jul 2013
Posts: 89
|
Better add the noindex metatag on your disallowed files. robots.txt is accessible to everyone and the users can see the disallowed files there.Or block the googlebot user agent from .htaccess
|
|
|
|
|
08-01-2013, 07:55 AM
|
#6 |
|
Guest
Posts: n/a
|
<meta name="robots" content="noindex, nofollow">
|
|
|
|
08-02-2013, 08:38 PM
|
#7 | |
|
Confirmed User
Industry Role:
Join Date: Nov 2012
Location: New Orleans
Posts: 213
|
Quote:
As far as Webmaster Tools, it wouldn't hurt to remove the links there as well, but I wouldn't do it instead of robots.txt (there are plenty of other sites that could potentially index the content). I would, however, upload your sitemap to Webmaster Tools if you haven't already.
__________________
Tent Pitcher - Adult Search Engine |
|
|
|
|
