[DoE] Open Energy Data Initiative - National Solar Radiation Database (OEDI-NSRD)

schoeneh · 2025-02-07T23:54:44.889Z

“I have 1998-2002 downloading now of the 5.7gb.”

schoeneh · 2025-02-07T23:55:05.007Z

“Now 03 and 04”

sneakers-the-rat · 2025-02-07T23:58:57.287Z

this is of the puerto rico datasets yes? the 5.7gb files?

schoeneh · 2025-02-07T23:59:19.100Z

yeah - will update the list (I can click the boxes on my own)

sneakers-the-rat · 2025-02-08T00:00:57.185Z

ok since the s3 downloader downloads all the files in parallel rather than sequentially, i’m going to go one by one so we at least have some useful data if the link gets shut off, going backwards in tdy from 2020. unchecking the rest and will check off as i go

Asta · 2025-02-08T00:06:19.615Z

I can do all of

831.6 GiB v3/tgy/nsrdb_tgy-2016.h5
831.6 GiB v3/tgy/nsrdb_tgy-2017.h5
796.3 GiB v3/tgy/nsrdb_tgy-2018.h5
796.3 GiB v3/tgy/nsrdb_tgy-2019.h5
796.3 GiB v3/tgy/nsrdb_tgy-2020.h5

if that works

I’ll start at the most recent (2020) and go backwards so if others want to take some just ping and I’ll report my progress

EDIT: estimate is probably around 12 hours for the tgy set

Asta · 2025-02-08T00:29:00.294Z

Friend of mine is doing v3/nsrdb_2019.h5 right now.

sneakers-the-rat · 2025-02-08T00:46:01.714Z

highest priority is apparently the tmy directory - so while someone is already on this, if anyone else has spare bandwidth and space, duplicates of that would be welcome

zyyygz · 2025-02-08T01:08:32.273Z

Unless there is something more important, I will grab a copy of tmy and:

  1.6 TiB v3/nsrdb_2012.h5
  1.6 TiB v3/nsrdb_2013.h5
  1.6 TiB v3/nsrdb_2014.h5
  1.6 TiB v3/nsrdb_2015.h5
  1.6 TiB v3/nsrdb_2016.h5
  1.6 TiB v3/nsrdb_2017.h5
  1.5 TiB v3/nsrdb_2018.h5
  1.5 TiB v3/nsrdb_2019.h5
  1.5 TiB v3/nsrdb_2020.h5

(assuming more recent data is more important)

Asta · 2025-02-08T01:38:10.730Z

Given the importance of tmy, I’ll grab v3/tmy/nsrdb_tmy-2018.h5 after nsrdb_tgy-2020.h5 finishes

Since it seems like people are generally going most recent to backwards.

EDIT since I’m limited to 3 replies lmao

Friend has switched from the above named one to v3/tmy/nsrdb_tmy-2017.h5 since that’s the critical set

EDIT AGAIN

I’ve finished tgy/nsrdb_tgy-2020.h5 and have moved on to tmy/nsrdb_tmy-2018.h5

irenes · 2025-02-08T02:50:24.387Z

I have people doing nsrdb_2018.h5, nsrdb_2017.h5, and tdy/nsrdb_tdy-2018.h5.

irenes · 2025-02-08T03:05:47.663Z

I also have someone doing nsrdb_2013.h5 but it’s slow

sneakers-the-rat · 2025-02-08T04:32:07.246Z

as we get close to the theoretical “end of day may mean midnight eastern time,” if everyone could take a second to refresh what they have and what they are grabbing that would be good:

for me:

Current status

file	completion	percent	rate	eta
`tdy/nsrdb_tdy-2018.h5`	431/796GB	54%	40MB/s	2h
`tdy/nsrdb_tdy-2019.h5`	494/796GB	62%	100MB/s	1h
`tdy/nsrdb_tdy-2020.h5`	645/796GB	81%	45MB/s	1h

Enqueued

I spend the day getting my NAS set up and had a bit of data loss from a canceled transfer, so i am currently only copying these single files, but after this round I will queue these up overnight, in case they stay up, going backwards from where @zyyygz is leaving off

1.6 TiB v3/nsrdb_2008.h5
1.6 TiB v3/nsrdb_2009.h5
1.6 TiB v3/nsrdb_2010.h5
1.6 TiB v3/nsrdb_2011.h5

and that’s probably all i’ll be able to grab overnight so i won’t claim more.

Question

Has anyone been able to exceed ~100MB/s on a single machine? I haven’t been able to with aws cli, rclone, or https downloading across single/multiple files and with a bunch of different threading and cache params. I have confirmed my download bandwidth is >500MB/s and my write speed is 1GB/s and can’t find the bottleneck unless it’s being imposed by AWS

irenes · 2025-02-08T06:40:37.611Z

from some testing a friend of mine did, it appears that Amazon rate-limits each downloading machine, across all download methods in use at the time

I do have to advise not trying to subvert this, it’s the kind of thing they could ban people for

Asta · 2025-02-08T08:53:33.099Z

Just finished tmy-2018; as I do not believe anyone has stated they have tmy-2019, I will download that before moving back to tgy.

zyyygz · 2025-02-08T09:00:26.870Z

tmy – 1.5TiB/4.0TiB – ETA 14h
nsrdb_2020.h5 1.3TiB/1.5TiB – ETA 1h
nsrdb_2018.h5 1.3TiB/1.5TiB – ETA 1h

skipping nsrdb_2019.h5 since it’s already checked off (?) and continuing with nsrdb_2017.h5 once 2018 is done.

does “checked” state mean “done”?

Has anyone been able to exceed ~100MB/s on a single machine?

above is on a single machine and appears to be 150MiB/s – three aws processes running in parallel (sync and two cp). no tuning. peak was 300MiB/s and it got throttled to this number. running a single aws process only gives me 90 MiB/s max.

stormaes · 2025-02-08T09:34:38.623Z

I’ve started on

1.6 TiB v3/nsrdb_1998.h5
1.6 TiB v3/nsrdb_1999.h5
1.6 TiB v3/nsrdb_2000.h5

Will post back once they’re done.

Jcryer · 2025-02-08T10:07:58.173Z

I have tmy-2016, I was getting ~45mb/s consistently overnight.

sneakers-the-rat · 2025-02-08T10:23:49.559Z

skipping nsrdb_2019.h5 since it’s already checked off (?)

yes yes claimed here

[DoE] Open Energy Data Initiative - National Solar Radiation Database (OEDI-NSRD) Needs Safeguarding 🚨

Friend of mine is doing v3/nsrdb_2019.h5 right now.

does “checked” state mean “done”?

sorry for lack of clarity here. checked state means someone has claimed it and they have either started downloading it or have it queued. it should be two columns, def will make this interface better on the site.

above is on a single machine and appears to be 150MiB/s – three aws processes running in parallel (sync and two cp). no tuning. peak was 300MiB/s and it got throttled to this number. running a single aws process only gives me 90 MiB/s max.

interesting. I started gathering some little snippets over here, and it sounds like we should have maybe a little tmux script that splits into multiple processes rather than letting the cli tool handle concurrency? i was looking in ToS for if this was an explicit limit, and it doesn’t seem like running downloaders in parallel would get us banned?

irenes · 2025-02-08T11:55:13.307Z

nsrdb_2018.h5 is now done at this end and nsrdb_2015.h5 is started (I am not the direct holder of this data, I’m just shepherding it)