Department of Education

markwyner · 2025-02-06T00:31:10.319Z

Hey, all.

I didn’t see edu.gov on our list. I’m currently scraping. It’s been working for some time now. I’ll let you know when I have it here and how it looks.

Thanks.

thgie · 2025-02-07T16:00:19.629Z

Any success? I also got all the files from https://ocrdata.ed.gov/data

AlRoeh · 2025-02-07T18:14:34.261Z

Hey,
I guess there is much more to this, but I’m looking into the ERIC fulltext db:

Fulltext Archive of the Department of Education / ERIC Database (eric.ed.gov) Needs Safeguarding 🚨

Still working on it. Already rescued 12,000+ files …

markwyner · 2025-02-07T19:27:54.337Z

It’s huge. I’ve had to pause/resume more than once. It’s still working. Currently at just over 2GB and ~3k files.

markwyner · 2025-02-07T21:52:02.382Z

Quick update. Well over 3.5GB now. Still going.

markwyner · 2025-02-08T01:02:36.753Z

13.5GB now. Almost 8k files. This heavy lift is PDFs. There are hundreds or maybe thousands of them. And they’re huge. I don’t think anyone using that system has any knowledge on how to compress a file.

Any advice from anyone on this? Should I just keep going? The main thing is that I have no idea how deep it goes. It could be a TB. Can you use a command line to get a directory storage size for a domain?

@schoeneh?

schoeneh · 2025-02-08T01:03:54.144Z

Have you got the storage capacity?
If yes, I say keep going

(Currently no idea re command line to get storage size)

markwyner · 2025-02-08T01:15:31.986Z

I have about 400GB. I can use about 300GB of that for this one site, then I’m stuck. And I’ll need to find a permanent home for it after that.

schoeneh · 2025-02-08T01:16:59.638Z

OK, that should be enough - those are PDFs, not datasets.
Don’t worry about the permanent home, we got things in the works.
(A few days would be enough)

Valjean · 2025-02-08T01:24:48.999Z

I have ~10tb available if you want me to make a copy. My internet is slow, though, and I can’t guarantee I won’t break something or lose files somehow, but I’m happy to hold a copy.

markwyner · 2025-02-08T01:34:58.343Z

Thanks, everyone. If I get to a dangerous place, I’ll pause and report back on my status. Then we can figure out what to do from there.