GoFuckYourself.com - Adult Webmaster Forum - View Single Post - BLOCKING crawlers whose SE's send you zero traffic

rowan · 10-20-2009, 06:25 PM

I've been looking at one of my mainstream sites and there are several crawlers (such as cuil.com's) that like to munch on my pages, but the search engine referral traffic they send is virtually (or even absolutely) zero.

There's a lot of pages on this site so a fair percentage of the access is from crawler bots. (GoogleBot hits it 100,000+ times a day but that figure is way ahead of the others, and G actually sends back referrals...)

I've been thinking about blocking the deadbeat crawlers via robots.txt, but then there's always the question looming - will the search engines they're attached to start sending traffic in the future? Am I going to shoot myself in the foot?

Has anyone contemplated this scenario?

10-20-2009, 06:25 PM
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	BLOCKING crawlers whose SE's send you zero traffic I've been looking at one of my mainstream sites and there are several crawlers (such as cuil.com's) that like to munch on my pages, but the search engine referral traffic they send is virtually (or even absolutely) zero. There's a lot of pages on this site so a fair percentage of the access is from crawler bots. (GoogleBot hits it 100,000+ times a day but that figure is way ahead of the others, and G actually sends back referrals...) I've been thinking about blocking the deadbeat crawlers via robots.txt, but then there's always the question looming - will the search engines they're attached to start sending traffic in the future? Am I going to shoot myself in the foot? Has anyone contemplated this scenario?