Quote:
Originally Posted by dazzling
Sphider looks good as a starting point for a smaller search engine before I try something bigger. Would it be comfortable with indexing 10,000 pages from say several hundred websites?
|
Try it and see, I do know that the sphider crawler is really slow as the data set grows.
A crawler really needs to be able to work quickly, ours are all written in C and are very nimble, not one line of code is included unless it is absolutely necessary for the crawler to work.