Originally Posted by rowan

PS: This phenomenon is also causing overloading issues because it's a database driven profile site... Googlebot is hitting me pretty much as fast as it can (around 85% of all fetches are from Googlebot), yet there seems to be no additional long term benefit from this. If the referral traffic scaled with the number of profile pages indexed then I'd just add more capacity, but it's not worth it.