Quote:
Originally Posted by plsureking
if they're not smart enough to check the reverse ip of a search bot, then they deserve to get ripped off. they have shitloads of overpaid engineers too.
i only allow authenticated search engine bots on our network. the rest of the bots, spiders, pirates, etc can fuck off. none of our sites need to be indexed in Zanzabar. most spiders are just making lists for pirates.
#
|
Yes but problem with some bots, bing for example, they offer some sort of public VPN/whatever which is used by attack bots, so you might thinking how it's legal because owner of ip is microsoft while in reality it's not, they just own ip. So you need to have exact ip ranges of search engine bot and whitelist it to make it work.