National Weather Service NEXRAD Level II data

XANTRONIX · 2025-02-06T18:00:51.998Z

NEXRAD Level II data is the reflectivity, velocity, and (since 2011) correlation coefficient weather radar data produced by the National Weather Service’s NEXRAD radar network across the CONUS and world, and archived by the National Centers for Environmental Information.

While the data is internally maintained in NCEI tape libraries, and available for order by the public through those systems, the data is also made available by Unidata in an Amazon S3 bucket: NEXRAD on AWS - Registry of Open Data on AWS

This information dates from 1991 onward, with dataset sizes increasing greatly:

/mnt/hdd/Archive$ du -sh *
11G	1991
65G	1992
172G	1993
346G	1994
2.2T	1995

Emphasis of this topic is placed on Level II data, as all the derived NEXRAD Level III radar products can be regenerated from this base data at a future time if need be. Estimates are that 2024 are about 31TiB. The data approaches petabyte scale.

schoeneh · 2025-02-06T18:45:38.133Z

ok - we can actually deal with PB-scale (kinda).
From which end should we start? From 2024 or 1991?

XANTRONIX · 2025-02-06T19:01:01.359Z

It’s hard to make a value judgement on this sort of thing but working backwards makes sense in terms of the greater frequency and intensity of certain types of storms.

XANTRONIX · 2025-02-06T19:14:46.584Z

Thinking about it, if you were to start from 2025 and go backwards, that would make sense; in case the S3 bucket were to go offline now, at least I would have 1991 through early 1996 locally.

schoeneh · 2025-02-06T23:14:52.052Z

Relaying information:
2025 is getting downloaded rn

zyyygz · 2025-02-07T18:12:57.809Z

I am starting with 2024 but it is unclear to me whether I have sufficient disk space. It seems to me that Jan and Feb of 2024 are more than 7.5 TiB together.

I am also downloading a full file listing using aws s3 ls --summarize --recursive s3://noaa-nexrad-level2/ --no-sign-request | zstd > listing.txt.zst so that the file size estimates can become more credible. That compressed text file is currently 3.7G compressed and still going.

zyyygz · 2025-02-08T00:50:35.556Z

the listing is downloaded. folder/prefix size per year:

1970	2257
1991	11122790578
1992	69301460492
1993	183226130557
1994	370222599416
1995	2332780302519
1996	4421590493359
1997	5263502940872
1998	4864202827518
1999	4615720158438
2000	4174928051914
2001	4540437088316
2002	4934228152935
2003	5434589035559
2004	6581272194129
2005	7097508351483
2006	6833014780124
2007	6938910362163
2008	10920191191436
2009	16411828330109
2010	17109661393721
2011	19396614930482
2012	30190490555018
2013	40902600625394
2014	47496514781414
2015	53643965248929
2016	48362823870964
2017	44035594652039
2018	53496197492534
2019	61271004858084
2020	62090714944294
2021	74708568807515
2022	74109038430331
2023	81205455712204
2024	82543957776372
2025	4570248576419

There are 326672611 individual files and they amount to 891132029899888 bytes (~892TB)

The detailed listing is 23GB uncompressed, 4GB compressed (if you need it let me know)

I’m only at 10TiB for 2024 and according to the file listing it might be close to 80 TiB.