Googlebot and vBulletin ~ Lens
 

<b>Brin:</b>

"<i>Don't forget to leave those session ID's out. </i>"

<b>Webmaster using vBulletin:</b>

"<i>BINGO! My vBulletin site had less than 20 links, then I dumped the session ID part of the URL, then it went to 1870 links after the dance a month ago, then the freshbot took it up to 3,550 during the lasty week and a half, and this dance took it to 7,410. If Google ever ends up giving me too many visitors then I might have to put the session ID back in! The Googlebot almost crashed it once while chomping away at the links!</i>"

Is gfy getting left out of the crawl intentionally, or incidentally?

ive wondered why ive NEVER seen gofuckyourself come up for much in a search engine.

Metpond and shit used to bomb google for keywords, but ive never came across GFY as a result.

you dont even get it when you do a search for <a href=http://www.google.com/search?sourceid=navclient&ie=UTF-8&oe=UTF-8&q=go+fuck+yourself>'go fuck yourself'</a>

...bump...

ya look at this even

http://www.google.com/search?hl=en&l...gofuckyourself

no description, no title, last link

robots.txt

it crashed my board a couple weeks ago..

it only takes a couple hundred of hits per minute until vbulletin clogs up the mysql..

Lensman doesn't want any pages listed on any search engine is my guess based on GFY's robots.txt file showing this:

User-agent: *
Disallow: /


WG

Would you want google to download millions of pages monthly with little to no return on it?

If you stick GFY in a search engine to crawl, it'll probably shoot up to PR 6-7 instantly.

robots.txt

User-agent: *
Disallow: /

Too bad vbulletin didn't abstract its db interface. mySql crappy non-ACID compliance finally turns its head.

notjoe:

what does that cost if you disallow images? a couple dozen gigs of transfer a month?

otoh i can see why lensman wouldn't want this board to get choked with boneheaded porn surfers.

eg metafilter doesn't even let people join anymore

vbulletin is way to harsh on mysql resources. every piece of document is created on the fly by requesting info from huge mysql tables. thats why this site is running on a quad xeon server..

btw mysql is only getting better.. with the new version it gets much closer to become ACID complient..

by the way, ACID complience doesnt mean much to me unless it is something as crucial as a banks account database.. it brings too much unneccessary overhead.

Why would anyone want GFY on a search engine?


We get enough type-in non adult webmaster idiots as it is...

Lots of things are mission critical. For example last year I wrote an auditing system for the Air Force/NSA encryption keys used for two-person controlled nuclear weapons.

The collaborative geospatial system I'm writing code for right now would probably break without solid transactional support, checkpoints, and rollbacks.

mysql is fast for serving dynamic web pages, questionable for highly interactive web services, and just plain wrong for anything mission critical.

cant beat the price though :thumbsup

Postgresql is also free and opensource, is acid compliant, and has some kickass OS stored procedure libraries.

i hope this is correct

User-agent: *
Allow: /
Disallow:

any idea how to leave the session id's out?

I'm not running vBulletin, but this thread might help:

http://www.webmasterworld.com/forum3...lletin+session

Our policy is "What's said here stays here."

We don't allow spiders and search engines for poster privacy reasons.

Nothing wrong with that in my book.

Why the hell would the rest of you guys want this site to be crawled? It wouldn't be so you can get free relevance for your spam links now would it? :1orglaugh

Good job Lens...keep this board "industry only" as much as possible.

that aint too bad.

maybe thats why SetGo seems to need to make copies of the 'good stuff'

hahaha

:glugglug

A blank disallow line like this actually means *allow* everything. Is this what you want? :) If so, it's probably redundant because of the allow line above it...

:glugglug