how to find the duplicate urls within tousands - GoFuckYourself.com

zentz · 05-08-2006, 11:17 PM

i have a .txt file with tousands of urls. now my script has reported that there are 20 duplicate urls but it cant tell which ones are they. is there a script or a software that can find and list the duplicates for me so i can fix them properly ?

pussyluver · 05-08-2006, 11:26 PM

Use the search function in notepad. Start at the top.

Study Visual Basic and write a program.

There's a couple of ideas. Hope someone has something a bit more direct.

latinasojourn · 05-09-2006, 12:09 AM

Links Suite 4

pudcat · 05-09-2006, 12:25 AM

if you know php or some other scripting language then it's pretty damn easy.

One method:

split it into an array
loop through the array
check if you've already seen that url, if not then remember it
print out a list of urls that you've seen

probbaly a cleaner way though I'm lazy

spunkmaster · 05-09-2006, 12:34 AM

Go to download.com and get one of the super free notepad programs
and do a search and you'll find them really fast.

darksoul · 05-09-2006, 12:42 AM

From shell:

Code:

sort file.txt|uniq -d

05-08-2006, 11:17 PM	#1
zentz Confirmed User Industry Role: Join Date: Nov 2003 Posts: 8,053	how to find the duplicate urls within tousands i have a .txt file with tousands of urls. now my script has reported that there are 20 duplicate urls but it cant tell which ones are they. is there a script or a software that can find and list the duplicates for me so i can fix them properly ? __________________ Programs that owe me money ---- Epassporte.com ~ $2700 \| Protraffic.com ~ $2600 \| XonDemand.com ~ $3000 Email: [email protected]

05-09-2006, 12:25 AM	#4
pudcat Confirmed User Join Date: Mar 2003 Posts: 1,169	if you know php or some other scripting language then it's pretty damn easy. One method: split it into an array loop through the array check if you've already seen that url, if not then remember it print out a list of urls that you've seen probbaly a cleaner way though I'm lazy __________________ SUBMIT YOUR BABE GALLERIES PROMOTE YOUR BLOG HERE always looking for hardlinks icq #207011694 Thunder-Ball.net, good for hardlink exchanges

05-09-2006, 12:34 AM	#5
spunkmaster Confirmed User Join Date: Jan 2004 Posts: 2,052	Go to download.com and get one of the super free notepad programs and do a search and you'll find them really fast. __________________

05-09-2006, 12:42 AM	#6
darksoul Confirmed User Join Date: Apr 2002 Location: /root/ Posts: 4,997	From shell: Code: sort file.txt\|uniq -d __________________ 1337 5y54\|)m1n: 157717888 BM-2cUBw4B2fgiYAfjkE7JvWaJMiUXD96n9tN Cambooth

05-08-2006, 11:26 PM	#2
pussyluver Clueless OleMan Join Date: Mar 2003 Location: ICQ - 169903487 Posts: 11,009	Use the search function in notepad. Start at the top. Study Visual Basic and write a program. There's a couple of ideas. Hope someone has something a bit more direct.

05-09-2006, 12:09 AM	#3
latinasojourn Confirmed User Join Date: Oct 2003 Posts: 3,191	Links Suite 4