|
|
|
|
||||
|
Welcome to the GoFuckYourself.com - Adult Webmaster Forum forums. You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features. Registration is fast, simple and absolutely free so please, join our community today! If you have any problems with the registration process or your account login, please contact us. |
 |
|
|||||||
| Discuss what's fucking going on, and which programs are best and worst. One-time "program" announcements from "established" webmasters are allowed. |
|
|
Thread Tools |
08-06-2023, 06:58 PM
|
#1 |
|
I help you SUCCEED
Industry Role:
Join Date: Nov 2003
Location: The Pearl of the Orient Seas
Posts: 32,195
|
Anyone know of a good sitemap detector and scraper?
See title
I need a free or paid tool that a) detects the sitemap url of a site b) scrapes the urls of a site off its sitemap |
|
|
|
08-06-2023, 07:46 PM
|
#2 |
|
Pay It Forward
Industry Role:
Join Date: Sep 2005
Location: Yo Mama House
Posts: 76,999
|
just paste mysite.com/sitemap.xml then scrape the good ones. yes i know a scraper...
__________________
TRUMP 2025 KEKAW!!! - The Laken Riley Act Is Law! DACA ENDED - SUPPORT AZ HCR 2060 52R - email: brassballz-at-techie.com |
|
|
|
|
08-07-2023, 11:01 AM
|
#3 | |
Industry Role:
Join Date: Aug 2006
Location: Little Vienna
Posts: 32,235
|
Quote:
|
|
|
|
|
|
08-07-2023, 11:24 AM
|
#4 | |
|
Pay It Forward
Industry Role:
Join Date: Sep 2005
Location: Yo Mama House
Posts: 76,999
|
Quote:
__________________
TRUMP 2025 KEKAW!!! - The Laken Riley Act Is Law! DACA ENDED - SUPPORT AZ HCR 2060 52R - email: brassballz-at-techie.com |
|
|
|
|
|
08-07-2023, 11:43 AM
|
#5 |
|
Pounding Googlebot
Industry Role:
Join Date: Aug 2002
Location: Canada
Posts: 34,452
|
Well, the first part should be as easy as going to domain.com/robots.txt and then finding the sitemap. You can then copy/paste it into this site which will extract all the links for you:
https://www.seowl.co/sitemap-extractor/ WG
__________________
I play with Google. |
|
|
|
|
08-07-2023, 09:19 PM
|
#6 |
|
I help you SUCCEED
Industry Role:
Join Date: Nov 2003
Location: The Pearl of the Orient Seas
Posts: 32,195
|
Thank you, guys!
WG - thanks for the awesome tool! |
|
|
|
|
08-08-2023, 06:18 AM
|
#7 |
|
Confirmed User
Join Date: Jan 2003
Posts: 960
|
Scrapbox has a plugin
|
|
|
|
|
08-08-2023, 03:45 PM
|
#8 | |
|
Pay It Forward
Industry Role:
Join Date: Sep 2005
Location: Yo Mama House
Posts: 76,999
|
Quote:
Scrape all the URL’s from websites using the Sitemap Scraper, included with the ability to do deep crawls.
__________________
TRUMP 2025 KEKAW!!! - The Laken Riley Act Is Law! DACA ENDED - SUPPORT AZ HCR 2060 52R - email: brassballz-at-techie.com |
|
|
|
|
|
08-08-2023, 07:15 PM
|
#9 | |
|
I help you SUCCEED
Industry Role:
Join Date: Nov 2003
Location: The Pearl of the Orient Seas
Posts: 32,195
|
Quote:
|
|
|
|
|
|
08-08-2023, 09:25 PM
|
#10 | |
|
Pay It Forward
Industry Role:
Join Date: Sep 2005
Location: Yo Mama House
Posts: 76,999
|
Quote:
__________________
TRUMP 2025 KEKAW!!! - The Laken Riley Act Is Law! DACA ENDED - SUPPORT AZ HCR 2060 52R - email: brassballz-at-techie.com |
|
|
|
|
