Getting all urls of a website / domain - GoFuckYourself.com

AdultSites · 05-02-2015, 02:24 PM

Is it possible to know about a url, if it is not linked to from any other place on the Internet, including search engine sites, or not? I am not talking about owners of the sites obviously, but any person, trying to get max number of urls of any site.

Is mapping the site by internal linking the only way to do it? I know it is also possible to get data from paid tools, scrape search engine results, and other things like that. I also know that, in general, number of pages that are not linked to from anywhere on the Internet, is low / very low.

I've been wondering if it is possible with some kind of server / Linux techniques, but it would probably require breaking in into the system of a site (hacking, if so, I am not talking about that).

EddyTheDog · 05-02-2015, 02:31 PM

I would have said mapping the site would be the easiest way to go - There are sites that will do it - It's done a lot by people who need a sitemap for the search engines...

xXXtesy10 · 05-02-2015, 02:51 PM

SilentKnight · 05-02-2015, 04:57 PM

Quote:

Originally Posted by AdultSites

Is it possible to know about a url, if it is not linked to from any other place on the Internet, including search engine sites, or not? I am not talking about owners of the sites obviously, but any person, trying to get max number of urls of any site.

Is mapping the site by internal linking the only way to do it? I know it is also possible to get data from paid tools, scrape search engine results, and other things like that. I also know that, in general, number of pages that are not linked to from anywhere on the Internet, is low / very low.

I've been wondering if it is possible with some kind of server / Linux techniques, but it would probably require breaking in into the system of a site (hacking, if so, I am not talking about that).

What's your purpose for this?

05-02-2015, 02:24 PM	#1
AdultSites Confirmed User Industry Role: Join Date: Feb 2015 Posts: 203	Getting all urls of a website / domain Is it possible to know about a url, if it is not linked to from any other place on the Internet, including search engine sites, or not? I am not talking about owners of the sites obviously, but any person, trying to get max number of urls of any site. Is mapping the site by internal linking the only way to do it? I know it is also possible to get data from paid tools, scrape search engine results, and other things like that. I also know that, in general, number of pages that are not linked to from anywhere on the Internet, is low / very low. I've been wondering if it is possible with some kind of server / Linux techniques, but it would probably require breaking in into the system of a site (hacking, if so, I am not talking about that).

05-02-2015, 02:51 PM	#3
xXXtesy10 Fakecoin Investor Industry Role: Join Date: Jul 2012 Location: New Delhi, IN Posts: 7,128	__________________ WARNING: Stay Away From Marlboroack aka aka Brandon Ackerman https://gfy.com/21169705-post8.html Donny Long is Felon, Stalker, Scammer & Coward http://www.ripoffreport.com/reports/...lon-int-761244

05-02-2015, 02:31 PM	#2
EddyTheDog Just Doing My Own Thing Industry Role: Join Date: Jan 2011 Location: London, Spain, New Zealand, GFY - Not Croydon... Posts: 25,035	I would have said mapping the site would be the easiest way to go - There are sites that will do it - It's done a lot by people who need a sitemap for the search engines...