Thread: Massive Hits From MSNbot
View Single Post
Old 02-07-2013, 12:34 AM  
Naughty
Confirmed User
 
Industry Role:
Join Date: Jul 2001
Location: Utopia
Posts: 6,482
Quote:
Originally Posted by HomerSimpson View Post
bing crawler...

http://stackoverflow.com/questions/4...ic-irregularly

add this to robots.txt

User-agent: msnbot
Crawl-delay: 1
I said that ;-)
__________________
seks.ai for sale - ping me
Naughty is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote