GoFuckYourself.com - Adult Webmaster Forum - View Single Post - Anyone know of a good sitemap detector and scraper?

WiredGuy · 08-07-2023, 11:43 AM

Well, the first part should be as easy as going to domain.com/robots.txt and then finding the sitemap. You can then copy/paste it into this site which will extract all the links for you:

https://www.seowl.co/sitemap-extractor/

WG

08-07-2023, 11:43 AM
WiredGuy Pounding Googlebot Industry Role: Join Date: Aug 2002 Location: Canada Posts: 34,452	Well, the first part should be as easy as going to domain.com/robots.txt and then finding the sitemap. You can then copy/paste it into this site which will extract all the links for you: https://www.seowl.co/sitemap-extractor/ WG __________________ I play with Google.