Googlebot and vBulletin ~ Lens - GoFuckYourself.com

fnet · 04-11-2003, 01:28 PM

Brin:

"Don't forget to leave those session ID's out. "

Webmaster using vBulletin:

"BINGO! My vBulletin site had less than 20 links, then I dumped the session ID part of the URL, then it went to 1870 links after the dance a month ago, then the freshbot took it up to 3,550 during the lasty week and a half, and this dance took it to 7,410. If Google ever ends up giving me too many visitors then I might have to put the session ID back in! The Googlebot almost crashed it once while chomping away at the links!"

Is gfy getting left out of the crawl intentionally, or incidentally?

Fletch XXX · 04-11-2003, 01:30 PM

ive wondered why ive NEVER seen gofuckyourself come up for much in a search engine.

Metpond and shit used to bomb google for keywords, but ive never came across GFY as a result.

you dont even get it when you do a search for <a href=http://www.google.com/search?sourceid=navclient&ie=UTF-8&oe=UTF-8&q=go+fuck+yourself>'go fuck yourself'</a>

fnet · 04-11-2003, 06:51 PM

...bump...

Plat · 04-11-2003, 06:57 PM

ya look at this even

http://www.google.com/search?hl=en&l...gofuckyourself

no description, no title, last link

extreme · 04-11-2003, 06:59 PM

robots.txt

Lane · 04-11-2003, 07:04 PM

it crashed my board a couple weeks ago..

it only takes a couple hundred of hits per minute until vbulletin clogs up the mysql..

WiredGuy · 04-11-2003, 07:05 PM

Lensman doesn't want any pages listed on any search engine is my guess based on GFY's robots.txt file showing this:

User-agent: *
Disallow: /

WG

notjoe · 04-11-2003, 07:16 PM

Would you want google to download millions of pages monthly with little to no return on it?

AdultNex · 04-11-2003, 07:33 PM

If you stick GFY in a search engine to crawl, it'll probably shoot up to PR 6-7 instantly.

foe · 04-11-2003, 07:42 PM

robots.txt

User-agent: *
Disallow: /

fnet · 04-11-2003, 07:51 PM

Quote:

Originally posted by Lane
it crashed my board a couple weeks ago..

it only takes a couple hundred of hits per minute until vbulletin clogs up the mysql..

Too bad vbulletin didn't abstract its db interface. mySql crappy non-ACID compliance finally turns its head.

notjoe:

what does that cost if you disallow images? a couple dozen gigs of transfer a month?

otoh i can see why lensman wouldn't want this board to get choked with boneheaded porn surfers.

eg metafilter doesn't even let people join anymore

Lane · 04-11-2003, 07:59 PM

Quote:

Originally posted by fnet

Too bad vbulletin didn't abstract its db interface. mySql crappy non-ACID compliance finally turns its head.

notjoe:

what does that cost if you disallow images? a couple dozen gigs of transfer a month?

otoh i can see why lensman wouldn't want this board to get choked with boneheaded porn surfers.

eg metafilter doesn't even let people join anymore

vbulletin is way to harsh on mysql resources. every piece of document is created on the fly by requesting info from huge mysql tables. thats why this site is running on a quad xeon server..

btw mysql is only getting better.. with the new version it gets much closer to become ACID complient..

by the way, ACID complience doesnt mean much to me unless it is something as crucial as a banks account database.. it brings too much unneccessary overhead.

Wizzo · 04-11-2003, 08:00 PM

Why would anyone want GFY on a search engine?

We get enough type-in non adult webmaster idiots as it is...

fnet · 04-11-2003, 08:43 PM

Quote:

Originally posted by Lane

by the way, ACID complience doesnt mean much to me unless it is something as crucial as a banks account database.. it brings too much unneccessary overhead.

Lots of things are mission critical. For example last year I wrote an auditing system for the Air Force/NSA encryption keys used for two-person controlled nuclear weapons.

The collaborative geospatial system I'm writing code for right now would probably break without solid transactional support, checkpoints, and rollbacks.

mysql is fast for serving dynamic web pages, questionable for highly interactive web services, and just plain wrong for anything mission critical.

foe · 04-11-2003, 08:46 PM

cant beat the price though

fnet · 04-11-2003, 08:51 PM

Quote:

Originally posted by foe
cant beat the price though

Postgresql is also free and opensource, is acid compliant, and has some kickass OS stored procedure libraries.

jimmyf · 04-11-2003, 08:53 PM

i hope this is correct

User-agent: *
Allow: /
Disallow:

slapass · 04-11-2003, 09:16 PM

any idea how to leave the session id's out?

fnet · 04-11-2003, 09:29 PM

Quote:

Originally posted by slapass
any idea how to leave the session id's out?

I'm not running vBulletin, but this thread might help:

http://www.webmasterworld.com/forum3...lletin+session

Lensman · 04-11-2003, 09:30 PM

Our policy is "What's said here stays here."

We don't allow spiders and search engines for poster privacy reasons.

Ludedude · 04-11-2003, 10:13 PM

Quote:

Originally posted by Lensman
Our policy is "What's said here stays here."

We don't allow spiders and search engines for poster privacy reasons.

Nothing wrong with that in my book.

Why the hell would the rest of you guys want this site to be crawled? It wouldn't be so you can get free relevance for your spam links now would it?

Good job Lens...keep this board "industry only" as much as possible.

Fletch XXX · 04-11-2003, 10:15 PM

Quote:

Originally posted by Lensman
Our policy is "What's said here stays here."

We don't allow spiders and search engines for poster privacy reasons.

that aint too bad.

maybe thats why SetGo seems to need to make copies of the 'good stuff'

hahaha

rowan · 04-11-2003, 10:34 PM

Quote:

Originally posted by jimmyf
i hope this is correct

User-agent: *
Allow: /
Disallow:

A blank disallow line like this actually means *allow* everything. Is this what you want?

If so, it's probably redundant because of the allow line above it...

Fletch XXX · 04-12-2003, 12:56 AM

04-11-2003, 01:28 PM	#1
fnet Confirmed User Join Date: Jul 2002 Posts: 1,721	Googlebot and vBulletin ~ Lens <b>Brin:</b> "<i>Don't forget to leave those session ID's out. </i>" <b>Webmaster using vBulletin:</b> "<i>BINGO! My vBulletin site had less than 20 links, then I dumped the session ID part of the URL, then it went to 1870 links after the dance a month ago, then the freshbot took it up to 3,550 during the lasty week and a half, and this dance took it to 7,410. If Google ever ends up giving me too many visitors then I might have to put the session ID back in! The Googlebot almost crashed it once while chomping away at the links!</i>" Is gfy getting left out of the crawl intentionally, or incidentally? __________________ the sound of one hand googlewhacking

04-11-2003, 01:30 PM	#2
Fletch XXX GFY HALL OF FAME DAMMIT!!! Join Date: Jan 2002 Location: that 504 Posts: 60,840	ive wondered why ive NEVER seen gofuckyourself come up for much in a search engine. Metpond and shit used to bomb google for keywords, but ive never came across GFY as a result. you dont even get it when you do a search for <a href=http://www.google.com/search?sourceid=navclient&ie=UTF-8&oe=UTF-8&q=go+fuck+yourself>'go fuck yourself'</a> __________________ Want an Android App for your tube, membership, or free site? Need banners or promo material? Hit us up (ICQ Fletch: 148841377) or email me fletchxxx at gmail.com - recent work - About me

04-11-2003, 06:51 PM	#3
fnet Confirmed User Join Date: Jul 2002 Posts: 1,721	...bump... __________________ the sound of one hand googlewhacking

04-11-2003, 06:57 PM	#4
Plat Confirmed User Industry Role: Join Date: Jan 2002 Location: Clearwater, Florida Posts: 2,680	ya look at this even http://www.google.com/search?hl=en&l...gofuckyourself no description, no title, last link __________________ Im fuckin retired

04-11-2003, 06:59 PM	#5
extreme Confirmed User Industry Role: Join Date: Oct 2002 Location: lalaland Posts: 2,120	robots.txt __________________ Browse and find new paysites with screenshot-preview Export FHGs/FLVs/Paysites by JSON, XML, CSV and Text Find FHGs/FLVs/Paysites from any given Category or Search-query

04-11-2003, 07:04 PM	#6
Lane Will code for food... Join Date: Apr 2001 Location: Buckeye, AZ Posts: 8,496	it crashed my board a couple weeks ago.. it only takes a couple hundred of hits per minute until vbulletin clogs up the mysql.. __________________

04-11-2003, 07:05 PM	#7
WiredGuy Pounding Googlebot Industry Role: Join Date: Aug 2002 Location: Canada Posts: 34,486	Lensman doesn't want any pages listed on any search engine is my guess based on GFY's robots.txt file showing this: User-agent: * Disallow: / WG __________________ I play with Google.

04-11-2003, 07:16 PM	#8
notjoe Confirmed User Industry Role: Join Date: May 2002 Location: Toronto, Canada Posts: 5,599	Would you want google to download millions of pages monthly with little to no return on it?

04-11-2003, 07:33 PM	#9
AdultNex Confirmed User Join Date: Feb 2003 Location: Bostonnnn Posts: 8,985	If you stick GFY in a search engine to crawl, it'll probably shoot up to PR 6-7 instantly.

04-11-2003, 07:42 PM	#10
foe Confirmed User Join Date: May 2002 Location: CT Posts: 5,246	robots.txt User-agent: * Disallow: /

04-11-2003, 08:00 PM	#13
Wizzo 2011 GFY Hall of Fame! Industry Role: Join Date: Nov 2000 Location: Back in Texas! Posts: 15,224	Why would anyone want GFY on a search engine? We get enough type-in non adult webmaster idiots as it is... __________________ Looking for Opportunity!

04-11-2003, 08:46 PM	#15
foe Confirmed User Join Date: May 2002 Location: CT Posts: 5,246	cant beat the price though

04-11-2003, 08:53 PM	#17
jimmyf OU812 Join Date: Feb 2001 Location: California Posts: 12,651	i hope this is correct User-agent: * Allow: / Disallow: __________________ Epic CashEpic Cash works for me Solar Cash Paysite Plugin Gallery of the day freesites,POTD,Gallery generator with free hosting

04-11-2003, 09:16 PM	#18
slapass Too lazy to set a custom title Join Date: Nov 2002 Location: Earth Posts: 14,625	any idea how to leave the session id's out?

04-11-2003, 09:30 PM	#20
Lensman GFY Chaperone Join Date: Jan 2001 Location: Adult.com Posts: 9,846	Our policy is "What's said here stays here." We don't allow spiders and search engines for poster privacy reasons. Last edited by Lensman; 04-11-2003 at 09:32 PM..

04-12-2003, 12:56 AM	#24
Fletch XXX GFY HALL OF FAME DAMMIT!!! Join Date: Jan 2002 Location: that 504 Posts: 60,840	__________________ Want an Android App for your tube, membership, or free site? Need banners or promo material? Hit us up (ICQ Fletch: 148841377) or email me fletchxxx at gmail.com - recent work - About me