Webscraping

schoeneh · 2025-02-03T14:56:04.051Z

A collection of tools and/or advice for webscraping

jenniferplusplus · 2025-02-03T02:29:28.208Z

Some tools that are likely to be useful:

grab-site, the archival web crawler used by archive-team

github.com

GitHub - ArchiveTeam/grab-site: The archivist's web crawler: WARC output,...

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

heritrix, the archival web crawler used by the Internet Archive

github.com

GitHub - internetarchive/heritrix3: Heritrix is the Internet Archive's open-source,...

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

claudinec · 2025-02-04T03:53:03.245Z

ArchiveBox: suite of Python tools including command-line and web interfaces, can be installed with Docker. Saves web pages and media in a variety of formats including WARC via wget.

Feed it a bunch of URLs, or install the Chrome/chromium extension (can be on a different machine) and archive as you browse.

schoeneh · 2025-02-04T08:27:10.794Z

Also very easy to set up via https://pikapods.com

But: This tool only archives up to level 1 (all links once removed), not any further.

markwyner · 2025-02-04T10:11:25.354Z

I was just gonna use wget. Is there an advantage to using one of these other tools instead?

schoeneh · 2025-02-04T10:12:13.552Z

wget is a great place to start - but it doesn’t pull the CSS etc. pp. - archivebox and Heritrix pull the whole websites including everything to recreate them 100%

schoeneh · 2025-02-04T10:17:14.761Z

wget
I’ve been using the following commands to scrape the sitemap(-s) and to download the HTML:

wget -qO- "https://www.arts.gov/sitemap.xml?page=1" | grep -Po "<loc>\K.+?(?=</loc>)" > urls1.txt
wget --convert-links --no-parent --wait=2 -i urls6.txt -x 2>&1 | tee arts.gov-sitemap1.log

There are better ways for sure, but that’s what I have been using.

markwyner · 2025-02-04T10:26:57.773Z

This is how I use wget:

wget -m -p -E -k -np https://domain.com

-m, --mirror
Enable recursion/time-stamps, set infinite recursion depth, and retain FTP directory listings

-p, --page-requisites
Get all images/resources needed to display as HTML

-E, --adjust-extension
Save HTML/CSS files with their extensions

-k, --convert-links
Convert links to point to local files

-np, --no-parent
Don’t ascend to the parent directory

schoeneh · 2025-02-04T10:40:43.325Z

this works so well!

markwyner · 2025-02-05T00:23:32.149Z

Yeah, I don’t recall where I found this. Someone else crafted the recipe and I’ve just stored it with an alias in my zsh file.

CuriousApe · 2025-02-05T15:11:37.720Z

The tools from Webrecorder are good for particularly tricky cases that need browser-assisted archiving: Webrecorder: Web Archiving for All e.g. Browsertrix

CuriousApe · 2025-02-05T15:14:17.089Z

There’s also the iipc/awesome-web-archiving: An Awesome List for getting started with web archiving which I’ve helped maintain over the years.

schoeneh · 2025-02-05T15:33:05.797Z

Very good having you here!
We will restructure this forum a bit and add your links to a short tutorial.
If you know of any good ones re webscraping - feel free to add them here!

claudinec · 2025-02-08T00:51:05.702Z

But: This tool only archives up to level 1 (all links once removed), not any further.

True (though it’s better than nothing); one solution if offered by the target site is, if there’s a sitemap (either XML or regular web page) or RSS feed, you can use that as the starting URL with depth=1.