I archived some of the climate stuff

Kuchenmampfer · 2025-02-07T00:06:28.035Z

Hello,
I just browsed a bit through the climate stuff and added things to my scraping. This resulted in 140GB of data from 15 domains and you can find and browse through it here. Unfortunately, I ran out of disk space, so these are probably incomplete and I can not put them into Already Safeguarded . However, I do think this is already a nice start. If you think, something is irrelevant, let me know, so I can delete it and archive something else. And if you need an rsync link or something else to get the full data, let me also know.

mattkuebrich · 2025-02-07T00:30:43.275Z

Wow, this is incredible, thank you!

Kuchenmampfer · 2025-02-07T08:33:05.823Z

For easier overview, the domains I have stuff from are the following:

hprcc.unl.edu/
mrcc.purdue.edu/
ocean.weather.gov/
prism.oregonstate.edu/
sercc.com/
stateclimate.org/
urbanoceanlab.org/
wrcc.dri.edu/
www.climatehubs.usda.gov/
www.cpc.ncep.noaa.gov/
www.drought.gov/
www.nhc.noaa.gov/
www.nrcs.usda.gov/
www.weather.gov/
www.wpc.ncep.noaa.gov/

schoeneh · 2025-02-07T12:26:44.661Z

how deep did you scrape?

Kuchenmampfer · 2025-02-07T18:13:58.976Z

I just used the standard wget command form the base domain everywhere. So it should have scraped everything that is linked somewhere and under the same domain right? Except that the downloads cancelled at some point because my disk was full.

schoeneh · 2025-02-07T20:46:01.538Z

your last point is a good sign :)