NERSC
DECK v1 simulation original model output has been archived on NERSC HPSS using zstash. See below for some examples on how to retrieve files using zstash. For a more information, refer to the zstash documentation. Please contact Zeshawn Shaheen (Unlicensed) for help with zstash. The table below lists output location for the various simulations. Note that some archives initially contained a few corrupted files. These problems have all been corrected.
Simulation | HPSS location | Approximate size | Failed files and tarballs | zstash check output | |||
---|---|---|---|---|---|---|---|
piControl |
Alternate location of repaired archives:
| 37 TB
| |||||
abrupt-4xCO2/home/t/tang30/2018/E3SM_simulations/20180215.DECKv1b_abrupt4xCO2.ne30_oEC.edison | New location of repaired archives:
| 12 TB |
| ||||
1pctCO2 | /home/g/golaz/2018/E3SM_simulations/20180215.DECKv1b_1pctCO2.ne30_oEC.edison | New location of repaired archives:
| 11 TB
| ||||
Historical H1 |
New location of repaired archives:
| 12 TB | |||||
Historical _H1-failed.txt | Historical H2 | /home/t/tang30/2018/E3SM_simulations/20180220.DECKv1b_ H2.ne30_oEC.edisonNew location of repaired archives: |
| 12 TB | |||
Historical _H2.txtHistorical H3 |
| 12 TB | |||||
Historical H4 |
| 12 TB |
| Historical H4 | Alternate location:
| 12 TB | |
No failures out of 49 tar files. | Historical _H4.txtHistorical H5 | /home/g/golaz/2018/E3SM_simulations/20180307.DECKv1b_H5.ne30_oEC.edison | 12 TB | No failures out of 50 tar files. | |||
AMIP A1 | /home/t/tang30/2018/E3SM_simulations/20180316.DECKv1b_A1.ne30_oEC.edison | 4 TB | No failures out of 15 tar files. | ||||
AMIP A1 1850allF |
| 5 TB | No failures out of 15 tar files. | ||||
AMIP A1 1850aeroF | /home/t/tang30/2018/E3SM_simulations/20180508.DECKv1b_A1_1850aeroF.ne30_oEC.edison | 5 TB | No failures out of 15 tar files. | AMIP_A1_1850aeroF.txt | |||
AMIP A2 | /home/t/tang30/2018/E3SM_simulations/20180622.DECKv1b_A2.ne30_oEC.edison | 4 TB | No failures out of 16 tar files. | ||||
AMIP A2 1850allF | /home/t/tang30/2018/E3SM_simulations/20180622.DECKv1b_A2_1850allF.ne30_oEC.edison | 4 TBNo failures out of 15 tar files. | |||||
AMIP _ A2 _1850allF.txtAMIP A2 1850aeroF | /home/t/tang30/2018/E3SM_simulations/20180622.DECKv1b_A2_1850aeroF.ne30_oEC.edison | 4 TBNo failures out of 15 tar files. | |||||
AMIP _A2_1850aeroF.txtAMIP A3 | /home/t/tang30/2018/E3SM_simulations/20180716.DECKv1b_A3.ne30_oEC.edison | 5 TB | No failures out of 20 tar files. | ||||
AMIP A3 1850allF | /home/t/tang30/2018/E3SM_simulations/20180716.DECKv1b_A3_1850allF.ne30_oEC.edison | 5 TB | No failures out of 19 tar files. | ||||
AMIP_A3_1850allF.txt | AMIP A3 1850aeroF | /home/t/tang30/2018/E3SM_simulations/20180716.DECKv1b_A3_1850aeroF.ne30_oEC.edison | 5 TBNo failures out of 19 tar files. | AMIP_A3_1850aeroF.txt | |||
Total | 161 TB | 18/634 = 2.839% tar files failed. |
Known bad data
Some users have reported errors when trying to extract from HPSS archive. The cause of the bad data is currently unknown; possibilities include a bug in zstash or data file being corrupted to/on HPSS.
...
- Could extract files; they look like netCDF files but md5 checksums do not match original ones.
...
- Could extract first file, but nothing else.
...
- Chris Golaz was able to successfully rescue contents of tar archive; md5 checksums matched original ones.
- Data available under:
/global/cscratch1/sd/golaz/ACME_simulations/rescue/20180302.DECKv1b_H3.ne30_oEC.edison
Retrieving data with zstash
...
Code Block | ||
---|---|---|
| ||
zstash extract --hpss=/home/g/golaz/2018/E3SM_simulations/repaired/20180129.DECKv1b_piControl.ne30_oEC.edison "*.cam.h0.00[3-6]?-??.nc" |
where
is the path on HPSS listed in the table above, and "/home/g/golaz/2018/E3SM_simulations/repaired/20180129.DECKv1b_piControl.ne30_oEC.edison
*.cam.h0.00[3-6]?-??.nc
" is a matching pattern to select all cam.h0 files between years 0030 and 0069, enclosing it in double quotes avoids shell level substitution.
...