Quote:
Originally Posted by rowan
My site has a web crawler that gathers information about each site and displays it on a profile page. It's 100% autonomous. The complainant is saying that his site is closed and he wants his profile page removed. I can delete it but the data will just appear again because most of the information gathered is based on external references (eg anchor text, referers) from other sites... the crawler will continue to find that.
|
Sounds like it's time to add a blacklist to your crawler.
