.htaccess for blocking bad bots
 

Im getting way too much crap from the bots
YandexBot
AhrefsBot
MJ12bot

Anyone successful block them with a .htaccess

i found this one below

SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</Limit>

Those bots are harmless anyway, why do you want to block them? The malicious ones don't identify themselves in the UA.

Thank you i will try it

Because im getting a few emails every day saying that mysql has crashed so i asked my host and they said its because of bots

I've used http://www.CloudFlare.com which seems to be used by more people but was more of a good CDN without much of anything being blocked out in terms of security as I tested the free and the $20/month Pro plan.

Currently using http://www.Incapsula.com free plan which is blocking more malicious shit out than the $20/month Pro plan from CloudFlare. Haven't tried a paid option here yet though and not sure I will as the free plan combined with a hardened server is working well. Currently have (444) domains running through them.

So if just looking for a popular CDN go CloudFlare.
If looking for a popular CDN with better security built in I'd go Incapsula. :2 cents:

Nice i will have a look

There you go, just find their user agents on the net and replace:

For well behaved bots, it's better to block them via robots.txt in your main web directory. Apart from the occasional re-fetch of robots.txt (to check if you've changed your mind) you shouldn't see them in your logs ever again

This is what i settled on

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} AhrefsBot [OR]
RewriteCond %{HTTP_USER_AGENT} MJ12bot [OR]
RewriteCond %{HTTP_USER_AGENT} Yandex [OR]
RewriteCond %{HTTP_USER_AGENT} SemrushBot
RewriteRule . - [F,L]