Search Engine script - GoFuckYourself.com

jimbaman · 01-24-2013, 03:17 AM

Hello,
I am looking for a search enigne script, like goole .. with the feature to check linksback and modifying the quality score for keyword result ..
e.g. if site has 10 linkbacks rise the position of search result ...

Any idea ?

Thankyou
Jimba

AdultKing · 01-24-2013, 03:19 AM

indexing how many sites ?

jimbaman · 01-24-2013, 03:26 AM

Quote:

Originally Posted by AdultKing

indexing how many sites ?

approx 1000 sites

riofromrio · 01-24-2013, 03:35 AM

Quote:

Originally Posted by AdultKing

indexing how many sites ?

would like to know a good one ... 500k sites should be indexed

AdultKing · 01-24-2013, 03:51 AM

1000 Sites: Nutch + Hadoop
500k sites: get a small server farm then Nutch + Hadoop

AdultKing · 01-24-2013, 03:55 AM

It's worth noting that Nutch is not trivial to deploy, so if you modified your requirements there are plenty of far simpler script solutions to simple search.

jimbaman · 01-24-2013, 04:09 AM

Quote:

Originally Posted by AdultKing

1000 Sites: Nutch + Hadoop
500k sites: get a small server farm then Nutch + Hadoop

explain better please ...
I've no idea of what is that, and what is used for ...

Thankyou

AdultKing · 01-24-2013, 04:19 AM

Unless you use Sphider http://www.sphider.eu or a similar script then implementing a search engine is non trivial.

What you want to do can't be achieved by Sphider because the indexing, weighting and ranking of sites based upon keywords and inbound links is a resource intensive task.

Sphider does not support ranking based on inbound links, however Nutch will as will some other open source search platforms.

Implementing such a thing is non trivial. You could develop your own solution and run it's crawler on Amazon EC2 instances to save having to invest in hardware, however you still need somewhere to house your database.

AdultKing · 01-24-2013, 04:25 AM

Here's something I posted on the subject two years ago in response to a similar question.

https://gfy.com/showpost.php?p=18075524&postcount=8

It's a bit out of date now, I'd lean toward Nutch these days.

Bogdan · 01-24-2013, 06:16 AM

Adult king you fuckin' rock!!!

Porndex · 01-24-2013, 06:35 AM

Do you know PHP and/or a language you can dev with on a server (Perl, C++)? It's pretty simple to get a crawler happening if you're apt with programming. Once you've got an "indexer" happening, the weighting and all that jazz is the fun part

The search engine isn't really the hard part to program, more the crawler if you ask me. Depends on what you're crawling and how you're displaying it too.

01-24-2013, 03:17 AM	#1
jimbaman Confirmed User Industry Role: Join Date: Jun 2003 Posts: 267	Search Engine script Hello, I am looking for a search enigne script, like goole .. with the feature to check linksback and modifying the quality score for keyword result .. e.g. if site has 10 linkbacks rise the position of search result ... Any idea ? Thankyou Jimba __________________ Skype & Email ME -

01-24-2013, 06:16 AM	#10
Bogdan So Fucking Banned Industry Role: Join Date: Dec 2012 Posts: 384	Adult king you fuckin' rock!!! Adult king you fuckin' rock!!!

01-24-2013, 06:35 AM	#11
Porndex Registered User Industry Role: Join Date: Oct 2012 Location: Germany Posts: 28	Do you know PHP and/or a language you can dev with on a server (Perl, C++)? It's pretty simple to get a crawler happening if you're apt with programming. Once you've got an "indexer" happening, the weighting and all that jazz is the fun part The search engine isn't really the hard part to program, more the crawler if you ask me. Depends on what you're crawling and how you're displaying it too. Last edited by Porndex; 01-24-2013 at 06:45 AM..

01-24-2013, 03:19 AM	#2
AdultKing Raise Your Weapon Industry Role: Join Date: Jun 2003 Location: Outback Australia Posts: 15,605	indexing how many sites ?

01-24-2013, 03:51 AM	#5
AdultKing Raise Your Weapon Industry Role: Join Date: Jun 2003 Location: Outback Australia Posts: 15,605	1000 Sites: Nutch + Hadoop 500k sites: get a small server farm then Nutch + Hadoop

01-24-2013, 03:55 AM	#6
AdultKing Raise Your Weapon Industry Role: Join Date: Jun 2003 Location: Outback Australia Posts: 15,605	It's worth noting that Nutch is not trivial to deploy, so if you modified your requirements there are plenty of far simpler script solutions to simple search.

01-24-2013, 04:19 AM	#8
AdultKing Raise Your Weapon Industry Role: Join Date: Jun 2003 Location: Outback Australia Posts: 15,605	Unless you use Sphider http://www.sphider.eu or a similar script then implementing a search engine is non trivial. What you want to do can't be achieved by Sphider because the indexing, weighting and ranking of sites based upon keywords and inbound links is a resource intensive task. Sphider does not support ranking based on inbound links, however Nutch will as will some other open source search platforms. Implementing such a thing is non trivial. You could develop your own solution and run it's crawler on Amazon EC2 instances to save having to invest in hardware, however you still need somewhere to house your database.

01-24-2013, 04:25 AM	#9
AdultKing Raise Your Weapon Industry Role: Join Date: Jun 2003 Location: Outback Australia Posts: 15,605	Here's something I posted on the subject two years ago in response to a similar question. https://gfy.com/showpost.php?p=18075524&postcount=8 It's a bit out of date now, I'd lean toward Nutch these days.