GoFuckYourself.com - Adult Webmaster Forum

GoFuckYourself.com - Adult Webmaster Forum (https://gfy.com/index.php)
-   Fucking Around & Business Discussion (https://gfy.com/forumdisplay.php?f=26)
-   -   Robots.txt VS meta robots INDEX (https://gfy.com/showthread.php?t=904541)

Kard63 05-10-2009 05:56 AM

Robots.txt VS meta robots INDEX
 
I have a robots text that disables the entire site for indexing

# Make changes for all web spiders
User-agent: *
Disallow: /

but I have robots meta data at the top of each page saying to index it. Which will happen? I don't want it to get indexed but I dont want to disable the plugin adding the robots meta data either.

Which one of these rules will trump the other?

Thanks in advance - and please don't guess.

Kard63 05-10-2009 06:20 AM

come on guys, I know all the smart people are working Sunday morning. Hook me up with the info please.

18teens 05-10-2009 06:24 AM

robots.txt

u-Bob 05-10-2009 08:52 AM

robots.txt
think about it for a sec :)

woj 05-10-2009 08:54 AM

they should follow robots.txt first

voa 05-10-2009 09:05 AM

I have no idea

MediaGuy 05-10-2009 09:09 AM

robots.txt is looked for first, and followed - so it won't bother hitting the metas if you no-index the site.

Yahoo! will take forever to index and list you if you don't have a robots.txt, for example, even if you have a robots meta tag.


All times are GMT -7. The time now is 08:21 PM.

Powered by vBulletin® Version 3.8.8
Copyright ©2000 - 2025, vBulletin Solutions, Inc.
©2000-, AI Media Network Inc123