GoFuckYourself.com - Adult Webmaster Forum

GoFuckYourself.com - Adult Webmaster Forum (https://gfy.com/index.php)
-   Fucking Around & Business Discussion (https://gfy.com/forumdisplay.php?f=26)
-   -   Anyone know of a good sitemap detector and scraper? (https://gfy.com/showthread.php?t=1367279)

$5 submissions 08-06-2023 06:58 PM

Anyone know of a good sitemap detector and scraper?
 
See title

I need a free or paid tool that a) detects the sitemap url of a site b) scrapes the urls of a site off its sitemap

brassmonkey 08-06-2023 07:46 PM

just paste mysite.com/sitemap.xml then scrape the good ones. yes i know a scraper...

Klen 08-07-2023 11:01 AM

Quote:

Originally Posted by brassmonkey (Post 23162992)
just paste mysite.com/sitemap.xml then scrape the good ones. yes i know a scraper...

Yep or go to google and type site:domain.com sitemap , that should do the trick as well.

brassmonkey 08-07-2023 11:24 AM

Quote:

Originally Posted by Klen (Post 23163225)
Yep or go to google and type site:domain.com sitemap , that should do the trick as well.

im guessing he has a list i think scrapebox can check the list. scraping what is a mystery.

WiredGuy 08-07-2023 11:43 AM

Well, the first part should be as easy as going to domain.com/robots.txt and then finding the sitemap. You can then copy/paste it into this site which will extract all the links for you:

https://www.seowl.co/sitemap-extractor/

WG

$5 submissions 08-07-2023 09:19 PM

Thank you, guys!

WG - thanks for the awesome tool!

money biz 08-08-2023 06:18 AM

Scrapbox has a plugin

brassmonkey 08-08-2023 03:45 PM

Quote:

Originally Posted by brassmonkey (Post 23163230)
im guessing he has a list i think scrapebox can check the list. scraping what is a mystery.

sitemap scraper: scrapebox

Scrape all the URL’s from websites using the Sitemap Scraper, included with the ability to do deep crawls.

$5 submissions 08-08-2023 07:15 PM

Quote:

Originally Posted by brassmonkey (Post 23163701)
sitemap scraper: scrapebox

Scrape all the URL’s from websites using the Sitemap Scraper, included with the ability to do deep crawls.

Scrapebox is the swiss army knife of SEO

brassmonkey 08-08-2023 09:25 PM

Quote:

Originally Posted by $5 submissions (Post 23163761)
Scrapebox is the swiss army knife of SEO

i talked to him this morning... he's a genius


All times are GMT -7. The time now is 03:02 PM.

Powered by vBulletin® Version 3.8.8
Copyright ©2000 - 2025, vBulletin Solutions, Inc.
©2000-, AI Media Network Inc123