|
|
|
|
||||
|
Welcome to the GoFuckYourself.com - Adult Webmaster Forum forums. You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features. Registration is fast, simple and absolutely free so please, join our community today! If you have any problems with the registration process or your account login, please contact us. |
 |
|
|||||||
| Discuss what's fucking going on, and which programs are best and worst. One-time "program" announcements from "established" webmasters are allowed. |
|
|
Thread Tools |
10-28-2003, 12:04 PM
|
#1 |
|
Confirmed User
Join Date: Oct 2001
Posts: 6,693
|
WhiteHouse.gov robots file
http://www.whitehouse.gov/robots.txt
Of course, I know it's there just so eventually when we take over the world, Google and Alta Vista will have no proof that that's what we planned, but I just found it interesting.
__________________
<a href="http://www.adultcontent.co.uk">Adult Content UK - Great British Content</a> |
|
|
|
10-28-2003, 12:07 PM
|
#2 |
|
HAL 9000
Industry Role:
Join Date: May 2001
Posts: 34,515
|
Disallow: /president/gallery/nudepics/gangbangs
Disallow: /president/gallery/nudepics/fisting Disallow: /president/gallery/nudepics/anal interesting  |
|
|
|
|
10-28-2003, 01:01 PM
|
#3 | |
|
Pounding Googlebot
Industry Role:
Join Date: Aug 2002
Location: Canada
Posts: 34,486
|
Quote:
__________________
I play with Google. |
|
|
|
|
|
10-28-2003, 01:45 PM
|
#4 |
|
Entrepreneur
Join Date: Oct 2002
Location: USA
Posts: 31,429
|
Great homeland security they got there.
What a joke.
__________________
 If you would like to develop your domains, you can lease inexpensive foreign laborfrom the leaders in the field at iWebmasters.com TO LOWER YOUR COSTS AND INCREASE YOUR PRODUCTION! *** *** *** *** *** *** *** *** *** *** *** *** Domains Adult News KRL's Newsletter Biz Tips Just Listed Domains
|
|
|
|
|
10-28-2003, 01:52 PM
|
#5 |
|
Confirmed User
Join Date: Jul 2001
Posts: 5,856
|
? i dont get it. haha
isnt robots.txt supposed to disallow files that actually exist? thus the reason for disallowing them to be spidered? i dont know my SE stuff... so... yeah im stupid :P
__________________
the revolution is coming. |
|
|
|
|
10-28-2003, 01:52 PM
|
#6 |
|
HAL 9000
Industry Role:
Join Date: May 2001
Posts: 34,515
|
fuckers,they removed them!
|
|
|
|
|
10-28-2003, 01:56 PM
|
#7 |
|
Confirmed User
Join Date: Jan 2002
Location: In the walls of your house.
Posts: 3,985
|
You can find all sorts of interesting things snooping around in robots.txt files. You just have to watch out for spider traps.
__________________
"Every normal man must be tempted, at times, to spit on his hands, hoist the black flag, and begin slitting throats." --H.L. Mencken |
|
|
|
|
10-28-2003, 02:07 PM
|
#8 |
|
Confirmed User
Join Date: Jun 2003
Location: Las Vegas, NV
Posts: 697
|
What bothers me about all this is that they are disallowing it now, because they were caught CHANGING official White House docs and articles after they had been published.
It happened when Bush declared combat operations in Iraq over. The headline read "Bush declares combat over in Iraq" and then they went back after the press was bad and changed it to read "Bush declares major combat over in Iraq". People caught it because of the cached page in Google and gave the administration shit for it. I can't be sure, but I think I read about some of the articles having their text rewritten and such. Want to talk about "revisionist history".... But more then that... why would the President want to "hide" official White House releases from the web searching public? |
|
|
|
Need more cowbell?