View Single Post
Old 10-02-2016, 06:20 PM  
rowan
Too lazy to set a custom title
 
Join Date: Mar 2002
Location: Australia
Posts: 17,393
For well behaved bots, it's better to block them via robots.txt in your main web directory. Apart from the occasional re-fetch of robots.txt (to check if you've changed your mind) you shouldn't see them in your logs ever again

Code:
User-agent: Yandex
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: AhrefsBot
Disallow: /
rowan is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote