1

I think this is the right category for this -

I’ll write a longer post later/keep this root post updated once i do, but for now…

code: sneakers-the-rat/sciop: collecting at-risk data in torrent rss feeds - Codeberg.org
url: (will post in a day or two once the site is more stable)

pitch:

  • No single archive can hold all the data
  • Even archive.org will eventually become a target of the administration, as will any recognizable archival group that are hosting/providing access to material that the government wants gone
  • Individual people have a lot of spare resources and want to help, and have been doing a lot of scraping on their own
  • There is not a good way of coordinating scraping work or indexing what has or has not been preserved in a way that can span archives

project summary:
Making a very lightweight indexing overlay to distribute archives files via bittorrent. Specifically, making a website that can

  • Accept requests for dataset archival
  • Index datasets that have been archived
  • Upload/Download .torrent files
  • Importantly, generate .rss feeds of .torrent files so that people with spare seedboxes can automatically help preserve data as it’s archived
  • Prioritize archiving requests by voting
  • Balance distributed hosting by showing which torrents have seeds and which need them
  • Have a dedicated API for mass uploading, and a dedicated mirroring system so that multiple such sites can exist and none of them becomes a single point of failure.

So far we’re at a very shaky MVP and will be continuing to work on it in the coming days, but it is at a state where it can accept contributions, so help wanted <3

3 Likes
2

Super awesome project, this is definitely something the web needed anyway

I’d be interested in how this compares with something like https://ipfs.tech/
-Turkey Can’t Block This Copy of Wikipedia | | Observer

I haven’t really looked into the technology much and it seems like it’s not very mature - but at least the goals seem somewhat similar

1 Like
3

i have wanted to love IPFS for as long as i have known it, but it just doesn’t really work all that well. for example this is the way you’re supposed to be access the most recent version according to that article and atm it’s not working, i’ve never had IPNS work: https://ipfs.io/ipns/k2k4r8lzhyyezzczuoirxvp4nhidd3wfehdtx5h1napkhz2jv7kpnwvv/wiki/Anasayfa.html

bittorrent just plain, flat out, works. + there are lots of people out there with seedboxes already established, and academic torrent trackers, etc. really the thing that thsi is gonna do is just be a coordination point, the organization is the thing that makes it really work <3

1 Like