.htaccess for blocking bad bots - GoFuckYourself.com

AllAboutCams · 10-01-2016, 09:23 AM

Im getting way too much crap from the bots
YandexBot
AhrefsBot
MJ12bot

Anyone successful block them with a .htaccess

i found this one below

SetEnvIfNoCase User-Agent .*sitebot.* bad_bot
<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</Limit>

Gallery-Traffic · 10-01-2016, 11:32 AM

Code:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^.*(AhrefsBot|MJ12bot|Yandex).*$ [NC]
RewriteRule .* - [F,L]

redwhiteandblue · 10-01-2016, 11:36 AM

Those bots are harmless anyway, why do you want to block them? The malicious ones don't identify themselves in the UA.

AllAboutCams · 10-01-2016, 09:10 PM

Quote:

Originally Posted by Gallery-Traffic

Code:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^.*(AhrefsBot|MJ12bot|Yandex).*$ [NC]
RewriteRule .* - [F,L]

Thank you i will try it

Quote:

Originally Posted by redwhiteandblue

Those bots are harmless anyway, why do you want to block them? The malicious ones don't identify themselves in the UA.

Because im getting a few emails every day saying that mysql has crashed so i asked my host and they said its because of bots

ErectMedia · 10-02-2016, 03:30 PM

I've used http://www.CloudFlare.com which seems to be used by more people but was more of a good CDN without much of anything being blocked out in terms of security as I tested the free and the $20/month Pro plan.

Currently using http://www.Incapsula.com free plan which is blocking more malicious shit out than the $20/month Pro plan from CloudFlare. Haven't tried a paid option here yet though and not sure I will as the free plan combined with a hardened server is working well. Currently have (444) domains running through them.

So if just looking for a popular CDN go CloudFlare.
If looking for a popular CDN with better security built in I'd go Incapsula.

AllAboutCams · 10-02-2016, 05:14 PM

Nice i will have a look

hdbuilder · 10-02-2016, 05:58 PM

There you go, just find their user agents on the net and replace:

Code:

# Block bad spiders
RewriteCond %{HTTP_USER_AGENT} Baiduspider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Sogou [NC,OR]
RewriteCond %{HTTP_USER_AGENT} GSLFbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Wotbox [NC,OR]
RewriteCond %{HTTP_USER_AGENT} MJ12bot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Sosospider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Ezooms [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Yandex
RewriteRule ^.* - [F,L]

rowan · 10-02-2016, 06:20 PM

For well behaved bots, it's better to block them via robots.txt in your main web directory. Apart from the occasional re-fetch of robots.txt (to check if you've changed your mind) you shouldn't see them in your logs ever again

Code:

User-agent: Yandex
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: AhrefsBot
Disallow: /

AllAboutCams · 10-02-2016, 06:42 PM

This is what i settled on

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} AhrefsBot [OR]
RewriteCond %{HTTP_USER_AGENT} MJ12bot [OR]
RewriteCond %{HTTP_USER_AGENT} Yandex [OR]
RewriteCond %{HTTP_USER_AGENT} SemrushBot
RewriteRule . - [F,L]

10-01-2016, 09:23 AM	#1
AllAboutCams Femcams.com Industry Role: Join Date: Jul 2011 Location: bjcam.com Posts: 12,220	.htaccess for blocking bad bots Im getting way too much crap from the bots YandexBot AhrefsBot MJ12bot Anyone successful block them with a .htaccess i found this one below SetEnvIfNoCase User-Agent .sitebot. bad_bot <Limit GET POST HEAD> Order Allow,Deny Allow from all Deny from env=bad_bot </Limit> __________________ Binance - Blockchain and Crypto Asset Exchange Chaturbate make money in cams

10-01-2016, 11:32 AM	#2
Gallery-Traffic Confirmed User Industry Role: Join Date: Mar 2015 Posts: 127	Code: RewriteEngine On RewriteCond %{HTTP_USER_AGENT} ^.(AhrefsBot\|MJ12bot\|Yandex).$ [NC] RewriteRule .* - [F,L] __________________ Gallery Traffic

10-01-2016, 11:36 AM	#3
redwhiteandblue Bollocks Industry Role: Join Date: Jun 2007 Location: Bollocks Posts: 2,792	Those bots are harmless anyway, why do you want to block them? The malicious ones don't identify themselves in the UA. __________________ Interserver unmanaged AMD Ryzen servers from $73.00

10-02-2016, 03:30 PM	#5
ErectMedia Confirmed Chicago Pimp Industry Role: Join Date: Aug 2004 Location: Chicago Posts: 7,100	I've used http://www.CloudFlare.com which seems to be used by more people but was more of a good CDN without much of anything being blocked out in terms of security as I tested the free and the $20/month Pro plan. Currently using http://www.Incapsula.com free plan which is blocking more malicious shit out than the $20/month Pro plan from CloudFlare. Haven't tried a paid option here yet though and not sure I will as the free plan combined with a hardened server is working well. Currently have (444) domains running through them. So if just looking for a popular CDN go CloudFlare. If looking for a popular CDN with better security built in I'd go Incapsula. __________________ Domain Name Registration Web3 Domains

10-02-2016, 05:14 PM	#6
AllAboutCams Femcams.com Industry Role: Join Date: Jul 2011 Location: bjcam.com Posts: 12,220	Nice i will have a look __________________ Binance - Blockchain and Crypto Asset Exchange Chaturbate make money in cams

10-02-2016, 05:58 PM	#7
hdbuilder Confirmed User Industry Role: Join Date: Jun 2012 Location: Canada Posts: 1,338	There you go, just find their user agents on the net and replace: Code: # Block bad spiders RewriteCond %{HTTP_USER_AGENT} Baiduspider [NC,OR] RewriteCond %{HTTP_USER_AGENT} Sogou [NC,OR] RewriteCond %{HTTP_USER_AGENT} GSLFbot [NC,OR] RewriteCond %{HTTP_USER_AGENT} Wotbox [NC,OR] RewriteCond %{HTTP_USER_AGENT} MJ12bot [NC,OR] RewriteCond %{HTTP_USER_AGENT} Sosospider [NC,OR] RewriteCond %{HTTP_USER_AGENT} Ezooms [NC,OR] RewriteCond %{HTTP_USER_AGENT} Yandex RewriteRule ^.* - [F,L] __________________ ROBO SCRIPTS \| WP CAM PLUGIN - Scripts To Promote Cam Sites - Chaturbate, BongaCams, Streamate, LiveJasmin, Stripchat... The Cam Site Builder, The Cam Multi Site Builder -> MULTIPLE CAM SITES IN ONE

10-02-2016, 06:20 PM	#8
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	For well behaved bots, it's better to block them via robots.txt in your main web directory. Apart from the occasional re-fetch of robots.txt (to check if you've changed your mind) you shouldn't see them in your logs ever again Code: User-agent: Yandex Disallow: / User-agent: MJ12bot Disallow: / User-agent: AhrefsBot Disallow: /

10-02-2016, 06:42 PM	#9
AllAboutCams Femcams.com Industry Role: Join Date: Jul 2011 Location: bjcam.com Posts: 12,220	This is what i settled on RewriteEngine On RewriteCond %{HTTP_USER_AGENT} AhrefsBot [OR] RewriteCond %{HTTP_USER_AGENT} MJ12bot [OR] RewriteCond %{HTTP_USER_AGENT} Yandex [OR] RewriteCond %{HTTP_USER_AGENT} SemrushBot RewriteRule . - [F,L] __________________ Binance - Blockchain and Crypto Asset Exchange Chaturbate make money in cams