Quote:
Originally Posted by Agent 488
if you are concerned about server stress from search engine bots you have bigger issues than the ones you may be contemplating.
|
Once again, there are more bots than humans loading this site. Out of ~200k loads per day roughly 60% are identifying themselves as bots, and there's probably 20-25% more that are rogue bots or site scrapers.
It's a profile site that pulls together various interlinking bits and pieces so it is reasonably database intensive.
So I'm not really concerned about server load NOW, more in the future... and really, I just don't get why I should let (eg) cuil scrape my site when they return zero traffic...