Business Best software that finds and filter bots ?

[PornoPlopedia]

What are the best software available to find and filter bot traffic ?
Thanks a lot

[PornoPlopedia]

bump for software that clean traffic

Here is a video describing my current state
https://www.youtube.com/watch?v=m_uWS6K-VF8

[Aka_Bluey]

Are you looking for a trade script?

Trade expert
Trade Expert - Free Traffic Trade Script - Trading Script - Traffic Trade - Trader Script

[PornoPlopedia]

Originally Posted by Aka_Bluey

Are you looking for a trade script?

Trade expert
Trade Expert - Free Traffic Trade Script - Trading Script - Traffic Trade - Trader Script

No I am looking more for something that detects fraudulent traffic off the bat. A script that can be used by anyone to detect patterns of bots based on an algorithm of logic. The same logic used by analysts when analyzing the traffic but automated.

[brandonstills]

What type of bots are you trying to prevent? Search engines, spiders, scrapers? Checking the host agent is the easiest but it is easy to fake. You can put some JS in your page that makes an outgoing request that tells you they are not a bot. That might block some. But a lot of bots now just use real browsers so that won't work. The next best approach is some kind of heuristics based approach that involves machine learning. That is really hard to determine and implement. It is also very specific to your site and will need a lot of fine tuning. Another simple answer is to just rate limit the number of requests per IP.

[PornoPlopedia]

Originally Posted by brandonstills

The next best approach is some kind of heuristics based approach that involves machine learning. That is really hard to determine and implement. It is also very specific to your site and will need a lot of fine tuning. Another simple answer is to just rate limit the number of requests per IP.

Brandonstills thanks for your smart answer.
A heuristic approach is exactly what I had in mind.

An algorithm to which you can feed a site's average KPIs to be used as the control group. Ideally set them as a range MIN MAX.

Example (not exhaustive) Takes into account #of requests by IP (vs Human Request)
AND User Agent
AND BR% range
AND Page/Sess
AND TOS
AND USER FLOW (1st/ 2nd/ nth interaction compared to the MIN MAX users habits from the same traffic type)
Etc.

It is a good project. The one that makes it can sell it to many, many webmasters/ traffic brokers..

[brandonstills]

Originally Posted by PornoPlopedia

Brandonstills thanks for your smart answer.
A heuristic approach is exactly what I had in mind.

An algorithm to which you can feed a site's average KPIs to be used as the control group. Ideally set them as a range MIN MAX.

That could work. The difference between a bot and a human would be that a bot would probably go through things faster. It might also go through the navigation slightly differently.

Are you trying to protect against people scraping from the site or just saving bandwidth? Even with IP rate limiting though, if they really wanted to they could just spin up 1000 different instances on Amazon Web Services and it would come from 1000 different IPs.

It all depends on how badly they want to crawl your site.

What's the nature of what you are protecting and what kind of threat are you trying to block? I might be able to give you a better answer if there is a more concrete example.

[Paz]

I had some friends who said the liked Bad Behaviour but I've never tried it myself;
Bad Behavior / Bad Behaviour | The Web's premier link spam killer.

[PornoPlopedia]

Originally Posted by brandonstills

What's the nature of what you are protecting and what kind of threat are you trying to block? I might be able to give you a better answer if there is a more concrete example.

It can have many functionalities
Qualifying Traffic by pretty much anyone when buying skimmed (and some new plugs) Monitoring the quality of that same traffic over time. Bcs you may receive a fair quality at first but then they can start sending fake clicks later.
Basically, need to be proactive instead of removing fake traffic from the stats system.
Thanks

[PornoPlopedia]

Originally Posted by Paz

I had some friends who said the liked Bad Behaviour but I've never tried it myself;
Bad Behavior / Bad Behaviour | The Web's premier link spam killer.

Thanks, I will look at this later.

[rowan]

Is CAPTCHA an option? I don't mean as the first line of defence (which is annoying for your users); rather, you challenge them only once your heuristic system suspects they may be a bot. There's a small chance of a human going down this road - say someone with cookies disabled, presenting no referer, coming via a proxy etc - but it's not really an issue since once they solve the CAPTCHA they can continue anyway (perhaps a successful CAPTCHA could also give them some positive heuristic score.) The more important thing is the accesses that do *not* solve the CAPTCHA, but just keep repeatedly hitting your challenge page without submitting a correct answer (or anything)... this further reinforces that it's some sort of automated agent accessing your site, not a human using a browser interactively.

I use this on a site which gets scraped to hell and back. I used to log headers and manually find patterns or signatures to block (most were pretty obvious) but an automated possible bot + CAPTCHA confirm system is so much easier.

[PornoPlopedia]

Originally Posted by rowan

Is CAPTCHA an option? I don't mean as the first line of defence (which is annoying for your users); rather, you challenge them only once your heuristic system suspects they may be a bot. There's a small chance of a human going down this road - say someone with cookies disabled, presenting no referer, coming via a proxy etc - but it's not really an issue since once they solve the CAPTCHA they can continue anyway (perhaps a successful CAPTCHA could also give them some positive heuristic score.) The more important thing is the accesses that do *not* solve the CAPTCHA, but just keep repeatedly hitting your challenge page without submitting a correct answer (or anything)... this further reinforces that it's some sort of automated agent accessing your site, not a human using a browser interactively.

I use this on a site which gets scraped to hell and back. I used to log headers and manually find patterns or signatures to block (most were pretty obvious) but an automated possible bot + CAPTCHA confirm system is so much easier.

Thanks for the tip ! Captcha is not an option tho in this case .

[brandonstills]

Originally Posted by PornoPlopedia

It can have many functionalities
Qualifying Traffic by pretty much anyone when buying skimmed (and some new plugs) Monitoring the quality of that same traffic over time. Bcs you may receive a fair quality at first but then they can start sending fake clicks later.
Basically, need to be proactive instead of removing fake traffic from the stats system.
Thanks

You might want to look into Complex Event Processing (CEP) solutions. I might set up some monitoring using something like Riemann - A network monitoring system.

[RazorSharpe]

Originally Posted by PornoPlopedia

Thanks for the tip ! Captcha is not an option tho in this case .

Block Bots | Bot Detection Distil Networks

[Bladewire]

Originally Posted by PornoPlopedia

What are the best software available to find and filter bot traffic ?
Thanks a lot

.htaccess can stop a lot of the low life bots,scrapers, etc. You get immediate results, though this took me a long time to test via trial and error. Here's a snippet from one of mine. A lot is cut out, this is only a snippet, immediately resolved some of my biggest issues. I allow more than the US on my sites, again, this is just a snippet and I've given just a few lines from each section

Options -indexes
ServerSignature Off
Options +FollowSymlinks
GeoIPEnable On
RewriteEngine On

SetEnvIf GEOIP_COUNTRY_CODE US AllowCountry
SetEnvIf GEOIP_COUNTRY_CODE CN AllowCountry
Allow from env=AllowCountry

# FORWARD CHINA TO A FITTING YOUTUBE VIDEO
<IfModule mod_rewrite.c>
RewriteCond %{ENV:GEOIP_COUNTRY_CODE} ^(CN)$
RewriteRule ^(.*)$ https://www.youtube.com/watch?v=SLMJpHihykI$1 [L] 
</IfModule>

# NO PROXIES FORWARDERS BLANK REFS ETC
RewriteCond %{HTTP:VIA}  !^$ [OR] 
RewriteCond %{HTTP:FORWARDED}  !^$ [OR] 
RewriteCond %{HTTP:USERAGENT_VIA}  !^$ [OR] 
RewriteCond %{HTTP:X_FORWARDED_FOR}  !^$ [OR] 
RewriteCond %{HTTP:PROXY_CONNECTION}  !^$ [OR] 
RewriteCond %{HTTP:XPROXY_CONNECTION}  !^$ [OR] 
RewriteCond %{HTTP:HTTP_PC_REMOTE_ADDR} !^$ [OR] 
RewriteCond %{HTTP:HTTP_CLIENT_IP}  !^$ 
# ISSUE 403 / SERVE ERRORDOCUMENT
RewriteRule ^(.*)$ - [F]

RewriteCond %{REQUEST_METHOD} POST
RewriteCond %{HTTP_REFERER} !.*YOURWEBSITE.COM* [OR]
RewriteCond %{HTTP_USER_AGENT} ^$
RewriteRule (.*) http://%{REMOTE_ADDR}/$ [R=301,L]

# STARTS WITH WEB
RewriteCond %{HTTP_USER_AGENT} ^web(zip|emaile|enhancer|fetch|go.?is|auto|bandit|clip|copier|master|reaper|sauger|site.?quester|whack) [NC]
 
# ANYWHERE IN UA -- GREEDY REGEX
RewriteCond %{HTTP_USER_AGENT} ^.*(craftbot|download|extract|stripper|sucker|ninja|clshttp|webspider|leacher|collector|grabber|webpictures).*$ [NC]
 
RewriteCond %{HTTP_USER_AGENT} ^.*(BlogScope|Butterfly|DCPbot|discoverybot|domain|Ezooms|ImageSearcherFree).*$ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*(ips-agent|linkdex|MJ12|Netcraft|NextGenSearchBot|SISTRIX|Sogou|soso|TweetmemeBot|Unwind|Yandex).*$ [NC]

RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]
RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR]
RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [OR]
RewriteCond %{HTTP_USER_AGENT} ^Navroad 
# ISSUE 403 / SERVE ERRORDOCUMENT
RewriteRule .* - [F,L]

# IF THE UA STARTS WITH THESE
RewriteCond %{HTTP_USER_AGENT} ^(aesop_com_spiderman|alexibot|backweb|bandit|batchftp|bigfoot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(black.?hole|blackwidow|blowfish|botalot|buddy|builtbottough|bullseye) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(cheesebot|cherrypicker|chinaclaw|collector|copier|copyrightcheck) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(cosmos|crescent|curl|custo|da|diibot|disco|dittospyder|dragonfly) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(drip|easydl|ebingbong|ecatch|eirgrabber|emailcollector|emailsiphon) [NC,OR]
# ISSUE 403 / SERVE ERRORDOCUMENT
RewriteRule .* - [F,L]

SetEnvIfNoCase User-Agent ^$ bad_bot
SetEnvIfNoCase User-Agent "^Download\ Demon" bad_bot
SetEnvIfNoCase User-Agent "^Download\ Devil" bad_bot
SetEnvIfNoCase User-Agent "^Download\ Wonder" bad_bot
SetEnvIfNoCase User-Agent "^dragonfly" bad_bot
SetEnvIfNoCase User-Agent "^Drip" bad_bot

# Vulnerability Scanners
SetEnvIfNoCase User-Agent "Acunetix" bad_bot
SetEnvIfNoCase User-Agent "FHscan" bad_bot
 
# Aggressive Chinese Search Engine
SetEnvIfNoCase User-Agent "Baiduspider" bad_bot
 
# Aggressive Russian Search Engine
SetEnvIfNoCase User-Agent "Yandex" bad_bot

<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</Limit>

[PornoPlopedia]

Originally Posted by Bladewire

.htaccess can stop a lot of the low life bots,scrapers, etc. You get immediate results, though this took me a long time to test via trial and error. Here's a snippet from one of mine. A lot is cut out, this is only a snippet, immediately resolved some of my biggest issues. I allow more than the US on my sites, again, this is just a snippet and I've given just a few lines from each section

Thanks a lot Bladewire ! Appreciated
Just sent a ticket to our developer to ad to our .htaccess

[PornoPlopedia]

Originally Posted by RazorSharpe

Block Bots | Bot Detection Distil Networks

Thanks RazorSharpe.
They seem to provide the service. I may go for the free trial.
If / when I do, I will keep you updated with the results

[PornoPlopedia]

Originally Posted by brandonstills

You might want to look into Complex Event Processing (CEP) solutions. I might set up some monitoring using something like Riemann - A network monitoring system.

Awesome, thanks brandonstills

Will check those out over the week end and see if we can implement this

[Honest-trafficker]

Pornoplopedia! I love how you're spanking those girls ;)

[freecartoonporn]

block bots at server level,
using firewall rules and rewrite rules.,
by checking user agents , and some other data.