View Single Post
Old 05-02-2010, 01:38 AM  
Davy
Confirmed User
 
Davy's Avatar
 
Industry Role:
Join Date: Apr 2006
Location: Germany
Posts: 4,323
Quote:
Originally Posted by rowan View Post
My site has a web crawler that gathers information about each site and displays it on a profile page. It's 100% autonomous. The complainant is saying that his site is closed and he wants his profile page removed. I can delete it but the data will just appear again because most of the information gathered is based on external references (eg anchor text, referers) from other sites... the crawler will continue to find that.
Sounds like it's time to add a blacklist to your crawler.
__________________
---
ICQ 14-76-98 <-- I don't use this at all
Davy is offline   Share thread on Digg Share thread on Twitter Share thread on Reddit Share thread on Facebook Reply With Quote