Quote:
Originally Posted by uno
Why don't you just use some sort of cacheing to cut down on DB queries?
|
With hundreds of millions of profiles it's pretty much all long tail, not much opportunity to cache anything. A page that a bot accesses may not be accessed by someone else for hours or even days. DB queries don't fare much better, with some basic profiling I've worked out that caching results would only result in a 5-10% reduction in raw queries, and at that level the negative overhead of caching could become comparable to the benefit it provides...
FWIW I am planning to set up multiple backend servers so that I can simply add more when the load gets too high... I'm really just curious whether anyone's said "fuck you" to a (bona fide but obscure) search engine bot that does nothing but scrape.
