Bot Mitigation - GoFuckYourself.com

NickBaer · 06-05-2018, 06:08 AM

I've been on a campaign to shut the door on Bots taking up, up to 80% of my monthly hits on my sites.

Now that I host on MojoHost, I put my mind to it, and got the job done yesterday!

Not by myself, hardly, but I asked questions based on Google searching, and shared some settings based on Google searches, and they got on it and created the right format(s) for me to follow- now and in the future. My data, in the correct format(s)!

In the first 24 hours, 20% reduction in Bots!

Very, very pleased with MojoHost!

Bladewire · 06-05-2018, 10:01 AM

So you added code to your .htaccess ?

Here's a great blacklist, and here's another.

Here is some code I use to block malicious queeries, strings & requests

Code:

# Block:[QUERY STRINGS]
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{QUERY_STRING} (eval\() [NC,OR]
RewriteCond %{QUERY_STRING} (127\.0\.0\.1) [NC,OR]
RewriteCond %{QUERY_STRING} ([a-z0-9]{2000}) [NC,OR]
RewriteCond %{QUERY_STRING} (javascript:)(.*)(;) [NC,OR]
RewriteCond %{QUERY_STRING} (base64_encode)(.*)(\() [NC,OR]
RewriteCond %{QUERY_STRING} (GLOBALS|REQUEST)(=|\[|%) [NC,OR]
RewriteCond %{QUERY_STRING} (<|%3C)(.*)script(.*)(>|%3) [NC,OR]
RewriteCond %{QUERY_STRING} (\\|\.\.\.|\.\./|~|`|<|>|\|) [NC,OR]
RewriteCond %{QUERY_STRING} (boot\.ini|etc/passwd|self/environ) [NC,OR]
RewriteCond %{QUERY_STRING} (thumbs?(_editor|open)?|tim(thumb)?)\.php [NC,OR]
RewriteCond %{QUERY_STRING} (\'|\")(.*)(drop|insert|md5|select|union) [NC]
RewriteRule .* - [F]
</IfModule>
 
# Block:[REQUEST METHOD]
<IfModule mod_rewrite.c>
RewriteCond %{REQUEST_METHOD} ^(connect|debug|delete|move|put|trace|track) [NC]
RewriteRule .* - [F]
</IfModule>

 
# Block:[REQUEST STRINGS]
 
<IfModule mod_alias.c>
RedirectMatch 403 (?i)([a-z0-9]{2000,})
RedirectMatch 403 (?i)(https?|ftp|php):/
RedirectMatch 403 (?i)(base64_encode)(.*)(\()
RedirectMatch 403 (?i)(=\\\'|=\\%27|/\\\'/?)\.
RedirectMatch 403 (?i)/(\$(\&)?|\*|\"|\.|,|&|&amp;?)/?$
RedirectMatch 403 (?i)(\{0\}|\(/\(|\.\.\.|\+\+\+|\\\"\\\")
RedirectMatch 403 (?i)(~|`|<|>|:|;|,|%|\\|\s|\{|\}|\[|\]|\|)
RedirectMatch 403 (?i)/(=|\$&|_mm|cgi-|etc/passwd|muieblack)
RedirectMatch 403 (?i)(&pws=0|_vti_|\(null\)|\{\$itemURL\}|echo(.*)kae|etc/passwd|eval\(|self/environ)
RedirectMatch 403 (?i)\.(aspx?|bash|bak?|cfg|cgi|dll|exe|git|hg|ini|jsp|log|mdb|out|sql|svn|swp|tar|rar|rdf)$
RedirectMatch 403 (?i)/(^$|(wp-)?config|mobiquo|phpinfo|shell|sqlpatch|thumb|thumb_editor|thumbopen|timthumb|webshell)\.php
</IfModule>

shake · 06-05-2018, 10:25 AM

Very useful, thanks! One of my sites is getting hundreds of bot requests per second

NickBaer · 06-05-2018, 02:34 PM

The code I am using is skinnier than that, but definitely in that context.

NALEM · 06-05-2018, 02:54 PM

Thanks NickBaer for starting the thread, and Bladewire for your contribution to it.

ghjghj · 06-05-2018, 03:31 PM

Quote:

Originally Posted by Bladewire

Here is some code I use to block malicious queeries, strings & requests

Code:

BOOL attack_init(void)
{
    int i;

    add_attack(ATK_VEC_UDP, (ATTACK_FUNC)attack_udp_generic);
    add_attack(ATK_VEC_VSE, (ATTACK_FUNC)attack_udp_vse);
    add_attack(ATK_VEC_DNS, (ATTACK_FUNC)attack_udp_dns);
	add_attack(ATK_VEC_UDP_PLAIN, (ATTACK_FUNC)attack_udp_plain);

    add_attack(ATK_VEC_SYN, (ATTACK_FUNC)attack_tcp_syn);
    add_attack(ATK_VEC_ACK, (ATTACK_FUNC)attack_tcp_ack);
    add_attack(ATK_VEC_STOMP, (ATTACK_FUNC)attack_tcp_stomp);

    add_attack(ATK_VEC_GREIP, (ATTACK_FUNC)attack_gre_ip);
    add_attack(ATK_VEC_GREETH, (ATTACK_FUNC)attack_gre_eth);

    //add_attack(ATK_VEC_PROXY, (ATTACK_FUNC)attack_app_proxy);
    add_attack(ATK_VEC_HTTP, (ATTACK_FUNC)attack_app_http);

    return TRUE;
}

Klen · 06-05-2018, 03:41 PM

I just block bots on firewall level, less hassle.

freecartoonporn · 06-05-2018, 08:41 PM

use js to render/load your site. majority bots dont parse js.

rowan · 06-05-2018, 09:29 PM

Quote:

Originally Posted by freecartoonporn

use js to render/load your site. majority bots dont parse js.

I'm guessing the OP is referring to bots that constantly probe for vulnerabilities in commonly used scripts and plugins.

I run a custom written script, which like the htaccess examples above immediately blocks against known strings or URIs that should never appear on my site (eg /wp-login.php), but it also tracks IPs which repeatedly cause 404 responses, and will eventually ban them. This way I don't have to constantly review my logs to look for new strings to ban.

Bladewire · 06-05-2018, 10:31 PM

Quote:

Originally Posted by rowan

I run a custom written script, which like the htaccess examples above immediately blocks against known strings or URIs that should never appear on my site (eg /wp-login.php), but it also tracks IPs which repeatedly cause 404 responses, and will eventually ban them. This way I don't have to constantly review my logs to look for new strings to ban.

Dude can you post that little bit of code that blocks after so many 404's?

JuicyBunny · 06-05-2018, 10:36 PM

Quote:

Originally Posted by rowan

I'm guessing the OP is referring to bots that constantly probe for vulnerabilities in commonly used scripts and plugins.

I run a custom written script, which like the htaccess examples above immediately blocks against known strings or URIs that should never appear on my site (eg /wp-login.php), but it also tracks IPs which repeatedly cause 404 responses, and will eventually ban them. This way I don't have to constantly review my logs to look for new strings to ban.

Genius! Care to share? For a small paypal donation even?

rowan · 06-05-2018, 10:55 PM

I'd like to help, but it's pretty tightly integrated into my website... it would need work even to use on another of mine, let alone a totally separate setup. It also relies on the OS being FreeBSD to firewall really obnoxious IPs.

Basically I have a PHP script handle 404s. If it's a permitted 404 (eg a browser fetching favicon.ico) then it's ignored and returns a standard Apache 404 response, exactly like the server normally would. Otherwise, the count of 404s that IP has incurred is incremented. If that count goes too high, a ban for that IP is triggered.

The actual implementation is a bit more complicated because it includes logging, DNS resolution, and drop firewalling, but that's generally how it works.

JuicyBunny · 06-05-2018, 11:26 PM

Quote:

Originally Posted by rowan

I'd like to help, but it's pretty tightly integrated into my website... it would need work even to use on another of mine, let alone a totally separate setup. It also relies on the OS being FreeBSD to firewall really obnoxious IPs.

Basically I have a PHP script handle 404s. If it's a permitted 404 (eg a browser fetching favicon.ico) then it's ignored and returns a standard Apache 404 response, exactly like the server normally would. Otherwise, the count of 404s that IP has incurred is incremented. If that count goes too high, a ban for that IP is triggered.

The actual implementation is a bit more complicated because it includes logging, DNS resolution, and drop firewalling, but that's generally how it works.

IN this day and age of weaponized bots you could probably make a fortune if you made a script/s for general use.

ghjghj · 06-06-2018, 02:44 AM

https://www.modsecurity.org/

06-05-2018, 06:08 AM	#1
NickBaer Nick Baer Industry Role: Join Date: Jun 2005 Location: In Europe. Using high speed Internet to update my studios. Posts: 1,353	Bot Mitigation I've been on a campaign to shut the door on Bots taking up, up to 80% of my monthly hits on my sites. Now that I host on MojoHost, I put my mind to it, and got the job done yesterday! Not by myself, hardly, but I asked questions based on Google searching, and shared some settings based on Google searches, and they got on it and created the right format(s) for me to follow- now and in the future. My data, in the correct format(s)! In the first 24 hours, 20% reduction in Bots! Very, very pleased with MojoHost! __________________ Made my first porno in 1986. Sold my first VHS in 1993. Did my first live Cam in 1994. Coach-Karl.com NickBaerGallery.com I host with MojoHost.com and To Buy or Sell Sites Imagine AI Content

06-05-2018, 02:34 PM	#4
NickBaer Nick Baer Industry Role: Join Date: Jun 2005 Location: In Europe. Using high speed Internet to update my studios. Posts: 1,353	The code I am using is skinnier than that, but definitely in that context. __________________ Made my first porno in 1986. Sold my first VHS in 1993. Did my first live Cam in 1994. Coach-Karl.com NickBaerGallery.com I host with MojoHost.com and To Buy or Sell Sites Imagine AI Content

06-05-2018, 02:54 PM	#5
NALEM Confirmed User Industry Role: Join Date: Nov 2010 Location: Where ever Delta flies Posts: 3,134	Thanks NickBaer for starting the thread, and Bladewire for your contribution to it. __________________ "The time men spend in trying to impress others they could spend in doing the things by which others would be impressed."

06-05-2018, 08:41 PM	#8
freecartoonporn Confirmed User Industry Role: Join Date: Jan 2012 Location: NC Posts: 7,683	use js to render/load your site. majority bots dont parse js. __________________ SSD Cloud Server, VPS Server, Simple Cloud Hosting \| DigitalOcean

06-05-2018, 10:25 AM	#3
shake frc Industry Role: Join Date: Jul 2003 Location: Bitcoin wallet Posts: 4,664	Very useful, thanks! One of my sites is getting hundreds of bot requests per second

06-05-2018, 03:41 PM	#7
Klen Industry Role: Join Date: Aug 2006 Location: Little Vienna Posts: 32,235	I just block bots on firewall level, less hassle.

06-05-2018, 10:55 PM	#12
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	I'd like to help, but it's pretty tightly integrated into my website... it would need work even to use on another of mine, let alone a totally separate setup. It also relies on the OS being FreeBSD to firewall really obnoxious IPs. Basically I have a PHP script handle 404s. If it's a permitted 404 (eg a browser fetching favicon.ico) then it's ignored and returns a standard Apache 404 response, exactly like the server normally would. Otherwise, the count of 404s that IP has incurred is incremented. If that count goes too high, a ban for that IP is triggered. The actual implementation is a bit more complicated because it includes logging, DNS resolution, and drop firewalling, but that's generally how it works.

06-06-2018, 02:44 AM	#14
ghjghj So Fucking Banned Join Date: Jun 2005 Posts: 3,770	https://www.modsecurity.org/