GoFuckYourself.com - Adult Webmaster Forum - View Single Post

rowan · 10-02-2016, 06:20 PM

For well behaved bots, it's better to block them via robots.txt in your main web directory. Apart from the occasional re-fetch of robots.txt (to check if you've changed your mind) you shouldn't see them in your logs ever again

Code:

User-agent: Yandex
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: AhrefsBot
Disallow: /

10-02-2016, 06:20 PM
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	For well behaved bots, it's better to block them via robots.txt in your main web directory. Apart from the occasional re-fetch of robots.txt (to check if you've changed your mind) you shouldn't see them in your logs ever again Code: User-agent: Yandex Disallow: / User-agent: MJ12bot Disallow: / User-agent: AhrefsBot Disallow: /