Originally Posted by AdultKing

Good question, there are search platforms suited for searching certain file types, for example mnGosearch was good at finding mp3 files, however scalability is a huge problem with that architecture, given that mapping the entire web and finding all mp3 files would be a herculean task.

Searching a group of sites is easier, if you know the sites you want to index then you're not having to discover the sites in the first place. If however you mean searching types of sites, this is the effort that our search engine is working on, simply indexing porn sites and almost nothing else. The crawling effort is huge, however the indexing effort is less of an issue as many sites are discarded as they are not porn sites.

Off the shelf is where things get difficult, because the differences in search efforts means that anything off the shelf needs a certain amount of customization.