NOAA Integrated Global Radiosonde Archive

XANTRONIX · 2025-02-06T21:52:16.339Z

From Integrated Global Radiosonde Archive (IGRA) | National Centers for Environmental Information (NCEI) :

The Integrated Global Radiosonde Archive (IGRA) consists of radiosonde and pilot balloon observations from more than 2,800 globally distributed stations. The earliest data date back to 1905, and recent data become available in near real time from about 800 stations worldwide. Observations are available at standard and variable pressure levels, fixed and variable-height wind levels, and the surface and tropopause. Variables include pressure, temperature, geopotential height, relative humidity, dew point depression, wind direction and speed, and elapsed time since launch.

The following NOAA National Centers for Environmental Information FTP endpoint provides an archive of global atmospheric soundings for the entire period of record:

ftp://ftp.ncei.noaa.gov/pub/data/igra/data/data-por/

This data amounts to 25GiB.

herrcaptain · 2025-02-07T00:06:26.556Z

Super new here - just got turned onto this group. I’m currently downloading this. Not sure what to do with the data from there, but I’ll keep it safe until someone needs it.
Is it just the contents of the /data-por/ directory that is needed? I see there’s also a data-ytd folder beside it in the tree.

schoeneh · 2025-02-07T00:13:26.810Z

You’re doing exactly what is needed right now
We’re working on the second part, don’t worry.

@XANTRONIX Can you take a look at the question here?

herrcaptain · 2025-02-07T00:45:04.143Z

Well, crap. I only got 76 out of the almost 3000 files in that FTP directory before they all started to fail with a permission error. When I reconnected to the server the data-por directory was gone. This was obviously just via an anonymous connection.

Hopefully someone else got it first.

XANTRONIX · 2025-02-07T00:58:34.156Z

The directory should still be present. I’m using ‘anonymous’ as the password, and of course ‘anonymous’ as the username.

image482×414 9.89 KB

Please let me know if you have any persistent issues with this and if so, I can help out.

You might need to tell wget or similar to back off and retry sometimes: wget -rc --no-parent --waitretry=10 --read-timeout=30 ftp://ftp.ncei.noaa.gov/pub/data/igra/data/data-por/

herrcaptain · 2025-02-07T01:16:40.738Z

Weird! You were correct - by reconnecting with those credentials it’s back up (though downloading super slowly, though could be my connection at the moment). I’m literally just using filezilla for this, so maybe that’s causing some of the issue. If it dies again I’ll try it via wget.

XANTRONIX · 2025-02-07T05:22:01.572Z

Apologies, @herrcaptain, I missed the original question! Just the /data-por/ directory should be fine; the data-ytd folder is only the most recent data.

herrcaptain · 2025-02-07T05:22:38.118Z

Head’s up that I’ve almost got this downloaded. Only about 400 MiB to go.
Being that this is only about 25 GiB of data, tomorrow I’ll make a second offsite backup and sit on it until ya’ll need it. It sounds like you @XANTRONIX are also grabbing it so hopefully between the two of us we’ll have complete copies of all the files just in case there was any corruption in either of our downloads.
This data is beyond my scope of understanding, but randomly opening up a few of the archives and looking at the contained .txt file shows human-readable data that appears to be intact.

XANTRONIX · 2025-02-07T05:25:05.448Z

Perfect! I’ve got a copy as well. The format of the sounding data is described here:

https://www.ncei.noaa.gov/data/integrated-global-radiosonde-archive/doc/igra2-data-format.txt

It would absolutely be worth capturing that near the top of the archive.

herrcaptain · 2025-02-07T05:30:57.435Z

Thanks so much for all the assistance. I’ve saved a copy of that txt file with the archive as requested.

chainik · 2025-02-07T10:42:33.656Z

XANTRONIX:

ftp://ftp.ncei.noaa.gov/pub/data/igra/data/data-por/

Mirrored here: igra

Torrent TODO but it would be great to have checksums from another copy from the source to check that these files are correct and not corrupted.

XANTRONIX · 2025-02-07T15:12:56.086Z

Happily! Here are the SHA512 sums of what I have:

https://xantronix.net/igra/SHA512SUMS

chainik · 2025-02-07T18:14:23.221Z

Hmmm… I get different numbers for many files (here are the first five):

==> SHA512.sums.files <==
29758971baaa0cd067d15d06972b2e50f5ac475745be353ef843f43829ebba1a6f61049bb80fdf914db2b5200e80d699132d4ca74056324e7572be1c8bda001a  ACM00078861-data.txt.zip
ef5f57766278f1f640f1674181ae349da1571e89943eb6d818517f52932fb743e6a635b8595152f3a95f38caef42169b5c53fb4bae548b7fd887506f643511a3  AEM00041217-data.txt.zip
06462af5d885588b1b3e006c924e08cff6971f3279fbfe811358bac3889720451a2c67cd0d63317aa29908a6657b758ad94decee46604fe2d79ba4b9cedbf3d0  AEXUAE05467-data.txt.zip
3a8608581a7b5eface8cac6c9575bec70f4b9e28d19882fdac7414462fa1d109f8571060a3c4400c5434d08ebf8bd94bcd263cd72336abe2bd65634c2eec5977  AFM00040911-data.txt.zip
9fd23dac59111a4b5007b553ca20ce3ea7fbc0b65ec648dd7a2e2d50f91072b554f10c9dfe761291bd3cc56e16d29c262405e3468e9a06a79115288baa6c95d3  AFM00040913-data.txt.zip

==> SHA512SUMS.sorted <==
7dc902780174e8d4d7584b4c6ffb20c64369a6918e156c8a35c5c1e240cdf60145aafe92250d6a8d8711525baf56777185c13f779055ad2a133ccd54e7c71619  ACM00078861-data.txt.zip
3f30ec303feee7efc7f3a8715d68ba2b2c9b06426e90acbf721370547f0202ec53f524b592dbb25f7299f2a0c41480fca7619a86f87e4be158d5cc3ac4c120d7  AEM00041217-data.txt.zip
5d9007f2009599770aecc4a2e13fc1dcff1387f051d47b9964a43884e7981943ea0674473ce32e1553652fde7a01101d4a7b8deafc268e2936ef8ae94442a838  AEXUAE05467-data.txt.zip
5214e35bae7745cb415db12274247661b1c8d76f2ff93fe027f9ae599519b8b14aee9ebb3472459565fc9a2f7906adad2bc4132bd684f365eaf7f61abfdb330c  AFM00040911-data.txt.zip
2fd87e8dcbeba468de06c52261c1e1bf6eed60ffbaecf6ab15461cba8916d7bdfb37a128023823c1a4afafec8e7cbd749006688a05d06f62997cf78d0bb10922  AFM00040913-data.txt.zip

do we have a third copy to check against to see which is likely to be wrong?

XANTRONIX · 2025-02-07T19:18:15.473Z

Since this is a live dataset, it is quite possible these changes simply reflect the latest readings. It might be worth a diff against two data files.

I’ve uploaded a sample for comparison:

https://xantronix.net/igra/ACM00078861-data.txt.zip

zyyygz · 2025-02-07T19:33:20.676Z

possibly relevant: @altnps.bsky.social on Bluesky

Elon Musk’s staffers have gained access to the computer systems at the National Oceanic and Atmospheric Administration (NOAA), claiming to use that access to search for programs and staff linked to diversity policies.

Simultaneously, sections of NOAA’s website displaying climate data have become unavailable. They are systematically going through the system, removing content at their discretion. More details to follow.

chainik · 2025-02-07T20:36:06.713Z

I do not think the data-por directory is live (I may be mistaken). The data-y2d directory is most likely live; y2d should mean year to date.

However, this is interesting. The data itself is the same but the zip files differ.

chainik@river:/tmp$ sha256sum a/ACM00078861-data.txt.zip b/ACM00078861-data.txt.zip 
dfdaab017f7d266455a2331211007390d38ddaa33fcd741a81033293f63586b4  a/ACM00078861-data.txt.zip
fc20cc2619cf283ed1612c4c6c5de7b351a455bc98e302a6ac372a9983000646  b/ACM00078861-data.txt.zip
chainik@river:/tmp$ sha256sum a/ACM00078861-data.txt b/ACM00078861-data.txt
b3a58c0b870adb50015290bc17376dc8f184c69c5bf9c20d4881bfd12a666318  a/ACM00078861-data.txt
b3a58c0b870adb50015290bc17376dc8f184c69c5bf9c20d4881bfd12a666318  b/ACM00078861-data.txt

chainik · 2025-02-07T20:38:32.808Z

The timestamps differ. Could it be the zip files are generated on the fly?

chainik@river:/tmp$ unzip -l a/ACM00078861-data.txt.zip 
Archive:  a/ACM00078861-data.txt.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
 27190992  2025-02-06 00:25   ACM00078861-data.txt
---------                     -------
 27190992                     1 file
chainik@river:/tmp$ unzip -l b/ACM00078861-data.txt.zip 
Archive:  b/ACM00078861-data.txt.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
 27190992  2025-02-07 00:24   ACM00078861-data.txt
---------                     -------
 27190992                     1 file

XANTRONIX · 2025-02-07T22:51:50.007Z

Ah yes; I think that might possibly be it. My knowledge of the NEXRAD dissemination process is way more involved than for the upper atmosphere data, but I expect the processes are documented with equal rigour. For now, I would consider our respective captures of the dataset to be trustworthy; I shall endeavour to find documentation on these processes.

Thank you very kindly for your diligence here!