Quote:
Originally posted by SpaceAce
The reason I ask is that the program I wrote to find expired domains takes a couple of hours to search for expired domains on seven major engines with a list of about 400 keywords/keyphrases when I have it set to go five pages deep. The time it ttakes includes performing all the whois operations and everything, but it is still a long time.
|
2 hours?! Have you thought about doing the crawl on a server, and getting people to subscribe to the data? You could also sell the processed data to other companies...
There's just something Not Quite Right about 1000 people running your application, all simultaneously raping the search engines for 2 hours straight... wouldn't it be better to do it once, and explode that data to your customers?
