Robots.txt help. How do I do this? - GoFuckYourself.com

wedouglas · 09-16-2006, 02:59 PM

I want to disallow the folder /sub/, but I want to allow everything in that folder. e.g. /sub/page1, /sub/page2

Any quick way to allow all files/folders inside /sub/ but disallow /sub/ alone?

After Shock Media · 09-16-2006, 03:06 PM

yeah but cant remember, damn pain meds.

but not all spiders follow those rules.

Umm I think it is
User-Agent: *
Disallow: /dirofyourdesire/

I take no responsibility if you loose all rankings, i am under the influence.

spasmo · 09-16-2006, 03:09 PM

Quote:

Originally Posted by After Shock Media

Umm I think it is
User-Agent: *
Disallow: /dirofyourdesire/

That will disallow all the dirs under it too.

Hope you feel better soon, man.

boldy · 09-16-2006, 03:09 PM

Why dont you remove the index.html /.php from /sub/
and add a .htaccess file in /sub/:
Options -Indexes

After Shock Media · 09-16-2006, 03:11 PM

Quote:

Originally Posted by spasmo

That will disallow all the dirs under it too.

Hope you feel better soon, man.

Thanks, i feel great actually. One of those things about pain meds.

Oh and thats also why i put in warning.

wedouglas · 09-16-2006, 03:14 PM

Quote:

Originally Posted by boldy

Why dont you remove the index.html /.php from /sub/
and add a .htaccess file in /sub/:
Options -Indexes

There isn't one. This script uses a lot of mod rewrite. I can't htacces http://site.com/sub/ because it doesn't actually exist. apparently I can't redirect it either. My only option is to disallow it.

borked · 09-16-2006, 03:19 PM

Quote:

Originally Posted by wedouglas

I want to disallow the folder /sub/, but I want to allow everything in that folder. e.g. /sub/page1, /sub/page2

Any quick way to allow all files/folders inside /sub/ but disallow /sub/ alone?

Disallow /sub
Allow /sub/

-edit...but to play it safe, just name all the folders to allow

Allow /sub/folder1
etc

borked · 09-16-2006, 03:24 PM

http://www.google.com/support/webmas...48&ctx=sibling

wedouglas · 09-16-2006, 03:37 PM

Quote:

Originally Posted by borked

http://www.google.com/support/webmas...48&ctx=sibling

Yeah, I use there tool to test it. I ended up just allowing /sub/a-z* and disallowing /sub/

Thanks

09-16-2006, 02:59 PM	#1
wedouglas So Fucking Banned Join Date: Aug 2004 Location: Dearborn, MI Posts: 5,921	Robots.txt help. How do I do this? I want to disallow the folder /sub/, but I want to allow everything in that folder. e.g. /sub/page1, /sub/page2 Any quick way to allow all files/folders inside /sub/ but disallow /sub/ alone?

09-16-2006, 03:06 PM	#2
After Shock Media It's coming look busy Join Date: Mar 2001 Location: "Ph'nglui mglw'nafh Cthulhu R'lyeh wgah'nagl fhtagn". Posts: 35,299	yeah but cant remember, damn pain meds. but not all spiders follow those rules. Umm I think it is User-Agent: * Disallow: /dirofyourdesire/ I take no responsibility if you loose all rankings, i am under the influence. __________________ [email protected] ICQ:135982156 AIM: Aftershockmed1a MSN: [email protected]

09-16-2006, 03:09 PM	#4
boldy Macdaddy coder Industry Role: Join Date: Feb 2002 Location: MacDaddy pimp coder Posts: 2,806	Why dont you remove the index.html /.php from /sub/ and add a .htaccess file in /sub/: Options -Indexes __________________ MacDaddy Coder.

09-16-2006, 03:24 PM	#8
borked Totally Borked Industry Role: Join Date: Feb 2005 Posts: 6,284	http://www.google.com/support/webmas...48&ctx=sibling __________________ For coding work - hit me up on andy // borkedcoder // com (consider figuring out the email as test #1) All models are wrong, but some are useful. George E.P. Box. p202