NEXRAD Level II data is the reflectivity, velocity, and (since 2011) correlation coefficient weather radar data produced by the National Weather Service’s NEXRAD radar network across the CONUS and world, and archived by the National Centers for Environmental Information.
While the data is internally maintained in NCEI tape libraries, and available for order by the public through those systems, the data is also made available by Unidata in an Amazon S3 bucket: NEXRAD on AWS - Registry of Open Data on AWS
This information dates from 1991 onward, with dataset sizes increasing greatly:
/mnt/hdd/Archive$ du -sh *
11G 1991
65G 1992
172G 1993
346G 1994
2.2T 1995
Emphasis of this topic is placed on Level II data, as all the derived NEXRAD Level III radar products can be regenerated from this base data at a future time if need be. Estimates are that 2024 are about 31TiB. The data approaches petabyte scale.
1 Like
ok - we can actually deal with PB-scale (kinda).
From which end should we start? From 2024 or 1991?
1 Like
It’s hard to make a value judgement on this sort of thing but working backwards makes sense in terms of the greater frequency and intensity of certain types of storms.
1 Like
Thinking about it, if you were to start from 2025 and go backwards, that would make sense; in case the S3 bucket were to go offline now, at least I would have 1991 through early 1996 locally.
1 Like
Relaying information:
2025 is getting downloaded rn
1 Like
zyyygz
6
I am starting with 2024 but it is unclear to me whether I have sufficient disk space. It seems to me that Jan and Feb of 2024 are more than 7.5 TiB together.
I am also downloading a full file listing using aws s3 ls --summarize --recursive s3://noaa-nexrad-level2/ --no-sign-request | zstd > listing.txt.zst so that the file size estimates can become more credible. That compressed text file is currently 3.7G compressed and still going.
1 Like
zyyygz
7
the listing is downloaded. folder/prefix size per year:
1970 2257
1991 11122790578
1992 69301460492
1993 183226130557
1994 370222599416
1995 2332780302519
1996 4421590493359
1997 5263502940872
1998 4864202827518
1999 4615720158438
2000 4174928051914
2001 4540437088316
2002 4934228152935
2003 5434589035559
2004 6581272194129
2005 7097508351483
2006 6833014780124
2007 6938910362163
2008 10920191191436
2009 16411828330109
2010 17109661393721
2011 19396614930482
2012 30190490555018
2013 40902600625394
2014 47496514781414
2015 53643965248929
2016 48362823870964
2017 44035594652039
2018 53496197492534
2019 61271004858084
2020 62090714944294
2021 74708568807515
2022 74109038430331
2023 81205455712204
2024 82543957776372
2025 4570248576419
There are 326672611 individual files and they amount to 891132029899888 bytes (~892TB)
The detailed listing is 23GB uncompressed, 4GB compressed (if you need it let me know)
I’m only at 10TiB for 2024 and according to the file listing it might be close to 80 TiB.
1 Like