21

I have tmy-2016, I was getting ~45mb/s consistently overnight.

2 Likes
22

skipping nsrdb_2019.h5 since it’s already checked off (?)

yes yes claimed here

does “checked” state mean “done”?

sorry for lack of clarity here. checked state means someone has claimed it and they have either started downloading it or have it queued. it should be two columns, def will make this interface better on the site.

above is on a single machine and appears to be 150MiB/s – three aws processes running in parallel (sync and two cp). no tuning. peak was 300MiB/s and it got throttled to this number. running a single aws process only gives me 90 MiB/s max.

interesting. I started gathering some little snippets over here, and it sounds like we should have maybe a little tmux script that splits into multiple processes rather than letting the cli tool handle concurrency? i was looking in ToS for if this was an explicit limit, and it doesn’t seem like running downloaders in parallel would get us banned?

1 Like
23

nsrdb_2018.h5 is now done at this end and nsrdb_2015.h5 is started (I am not the direct holder of this data, I’m just shepherding it)

1 Like
24

Plugging away at v3/nsrdb_2020.h5 largely done. Hopefully in the next few hours.