GoFuckYourself.com - Adult Webmaster Forum - View Single Post - Caffeine spiders my Mainstream sites in seconds

rowan · 06-19-2010, 03:25 AM

How could G consistently find new posts within 10 seconds unless it's repeatedly hammering a site? Imagine if all SE bots went fishing like that!

I kind of have the opposite problem, there's too much content for it to ever completely index so it just fetches endlessly, more than 120,000+ per day...

06-19-2010, 03:25 AM
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	How could G consistently find new posts within 10 seconds unless it's repeatedly hammering a site? Imagine if all SE bots went fishing like that! I kind of have the opposite problem, there's too much content for it to ever completely index so it just fetches endlessly, more than 120,000+ per day...