Welcome to the GoFuckYourself.com - Adult Webmaster Forum forums.

You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features. Registration is fast, simple and absolutely free so please, join our community today!

If you have any problems with the registration process or your account login, please contact us.

Post New Thread Reply

Register GFY Rules Calendar
Go Back   GoFuckYourself.com - Adult Webmaster Forum > >
Discuss what's fucking going on, and which programs are best and worst. One-time "program" announcements from "established" webmasters are allowed.

 
Thread Tools
Old 05-10-2009, 05:56 AM   #1
Kard63
Confirmed User
 
Kard63's Avatar
 
Join Date: Nov 2003
Location: 237 619 975
Posts: 8,944
Robots.txt VS meta robots INDEX

I have a robots text that disables the entire site for indexing

# Make changes for all web spiders
User-agent: *
Disallow: /

but I have robots meta data at the top of each page saying to index it. Which will happen? I don't want it to get indexed but I dont want to disable the plugin adding the robots meta data either.

Which one of these rules will trump the other?

Thanks in advance - and please don't guess.
__________________
Kard63 is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 05-10-2009, 06:20 AM   #2
Kard63
Confirmed User
 
Kard63's Avatar
 
Join Date: Nov 2003
Location: 237 619 975
Posts: 8,944
come on guys, I know all the smart people are working Sunday morning. Hook me up with the info please.
__________________
Kard63 is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 05-10-2009, 06:24 AM   #3
18teens
Confirmed User
 
Industry Role:
Join Date: Dec 2002
Posts: 1,605
robots.txt
18teens is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 05-10-2009, 08:52 AM   #4
u-Bob
there's no $$$ in porn
 
u-Bob's Avatar
 
Industry Role:
Join Date: Jul 2005
Location: icq: 195./568.-230 (btw: not getting offline msgs)
Posts: 33,063
robots.txt
think about it for a sec
u-Bob is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 05-10-2009, 08:54 AM   #5
woj
<&(©¿©)&>
 
woj's Avatar
 
Industry Role:
Join Date: Jul 2002
Location: Chicago
Posts: 47,882
they should follow robots.txt first
__________________
Custom Software Development, email: woj#at#wojfun#.#com to discuss details or skype: wojl2000 or gchat: wojfun or telegram: wojl2000
Affiliate program tools: Hosted Galleries Manager Banner Manager Video Manager
Wordpress Affiliate Plugin Pic/Movie of the Day Fansign Generator Zip Manager
woj is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 05-10-2009, 09:05 AM   #6
voa
Too lazy to set a custom title
 
Join Date: Nov 2006
Posts: 16,532
I have no idea
voa is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Old 05-10-2009, 09:09 AM   #7
MediaGuy
Confirmed User
 
MediaGuy's Avatar
 
Industry Role:
Join Date: Sep 2004
Location: Montrealquebecanada
Posts: 5,500
robots.txt is looked for first, and followed - so it won't bother hitting the metas if you no-index the site.

Yahoo! will take forever to index and list you if you don't have a robots.txt, for example, even if you have a robots meta tag.
__________________

YOU Are Industry News!
Press Releases: pr[at]payoutmag.com
Facebook: Payout Magazine! Facebook: MIKEB!
ICQ: 248843947
Skype: Mediaguy1
MediaGuy is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote
Post New Thread Reply
Go Back   GoFuckYourself.com - Adult Webmaster Forum > >

Bookmarks



Advertising inquiries - marketing at gfy dot com

Contact Admin - Advertise - GFY Rules - Top

©2000-, AI Media Network Inc



Powered by vBulletin
Copyright © 2000- Jelsoft Enterprises Limited.