Versions Compared

Old Version 26

changes.mady.by.user Zeshawn Shaheen (Unlicensed)

Saved on

compared with

New Version 27

changes.mady.by.user Chris Golaz

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

NERSC

DECK v1 simulation original model output has been archived on NERSC HPSS using zstash. See below for some examples on how to retrieve files using zstash. For a more information, refer to the zstash documentation. Please contact Zeshawn Shaheen (Unlicensed) for help with zstash. The table below lists output location for the various simulations. Note that some archives initially contained a few corrupted files. These problems have all been corrected.

SimulationHPSS locationApproximate sizeFailed files and tarballszstash check output
piControl


/home/g/golaz/2018/E3SM_simulations/repaired/20180129.DECKv1b_piControl.ne30_oEC.edisonNew

Alternate location

of repaired archives

:

/home/

g

t/

golaz

tang30/2018/E3SM_simulations/repaired/20180129.DECKv1b_piControl.ne30_oEC.edison

37 TB

piControl-failed.txt
00000a.tar
000074.tar
000029.tar
000081.tar

Total of 147 tar files

piControl.txt

abrupt-4xCO2/home/t/tang30/2018/E3SM_simulations/20180215.DECKv1b_abrupt4xCO2.ne30_oEC.edison

New location of repaired archives:

/home/g/golaz/2018/E3SM_simulations/repaired/20180215.DECKv1b_abrupt4xCO2.ne30_oEC.edison

12 TB

abrupt-4xCO2-failed.txt
00002a.tar
00001d.tar
00000f.tar
000028.tar
000021.tar

Total of 47 tar files

abrupt-4xCO2.txt

1pctCO2/home/g/golaz/2018/E3SM_simulations/20180215.DECKv1b_1pctCO2.ne30_oEC.edison

New location of repaired archives:

/home/g/golaz/2018/E3SM_simulations/repaired/20180215.DECKv1b_1pctCO2.ne30_oEC.edison

11 TB

1pctCO2-failed.txt
000000.tar

Total of 45 tar files

1pctCO2.txt

Historical H1

/home/t/tang30/2018/E3SM_simulations/20180215.DECKv1b_H1.ne30_oEC.edison

New location of repaired archives:

/home/g/golaz/2018/E3SM_simulations/repaired/20180215.DECKv1b_H1.ne30_oEC.edison

12 TB
Historical _H1-failed.txt
00001e.tar
000002.tarTotal of 49 tar files

Historical_H1.txt

Historical H2/home/t/tang30/2018/E3SM_simulations/20180220.DECKv1b_ H2.ne30_oEC.edisonNew location of repaired archives:

/home/g/golaz/2018/E3SM_simulations/repaired/20180220.DECKv1b_H2.ne30_oEC.edison

12 TBHistorical_H2-failed.txt000014.tar
00002d.tarTotal of 49 tar files
Historical _H2.txtHistorical H3

/home/gz/golazzshaheen/2018/E3SM_simulations/repaired/20180302.DECKv1b_H3.ne30_oEC.edisonNew location of repaired archives:

12 TB
Historical H4

/home/

z

t/

zshaheen

tang30/2018/E3SM_simulations/

repaired/20180302

20180305.DECKv1b_

H3

H4.ne30_oEC.edison

12 TB

Historical_H3-failed.txt
000000.tar
00002a.tar
00001f.tar
000025.tar

Total of 49 tar files

Historical_H3.txt

Historical H4

Alternate location:

/home/

t

g/

tang30

golaz/2018/E3SM_simulations/20180305.DECKv1b_H4.ne30_oEC.edison

12 TB

No failures out of 49 tar files.

Historical _H4.txtHistorical H5/home/g/golaz/2018/E3SM_simulations/20180307.DECKv1b_H5.ne30_oEC.edison12 TBNo failures out of 50 tar files.

Historical_H5.txt

AMIP A1/home/t/tang30/2018/E3SM_simulations/20180316.DECKv1b_A1.ne30_oEC.edison4 TBNo failures out of 15 tar files.

AMIP_A1.txt

AMIP A1 1850allF

/home/t/tang30/2018/E3SM_simulations/20180508.DECKv1b_A1_1850allF.ne30_oEC.edison

5 TBNo failures out of 15 tar files.

AMIP_A1_1850allF.txt

AMIP A1 1850aeroF/home/t/tang30/2018/E3SM_simulations/20180508.DECKv1b_A1_1850aeroF.ne30_oEC.edison5 TBNo failures out of 15 tar files.AMIP_A1_1850aeroF.txt
AMIP A2/home/t/tang30/2018/E3SM_simulations/20180622.DECKv1b_A2.ne30_oEC.edison4 TBNo failures out of 16 tar files.

AMIP_A2.txt

AMIP A2 1850allF/home/t/tang30/2018/E3SM_simulations/20180622.DECKv1b_A2_1850allF.ne30_oEC.edison4 TBNo failures out of 15 tar files.
AMIP _ A2 _1850allF.txtAMIP A2 1850aeroF/home/t/tang30/2018/E3SM_simulations/20180622.DECKv1b_A2_1850aeroF.ne30_oEC.edison4 TBNo failures out of 15 tar files.
AMIP _A2_1850aeroF.txtAMIP A3/home/t/tang30/2018/E3SM_simulations/20180716.DECKv1b_A3.ne30_oEC.edison5 TBNo failures out of 20 tar files.

AMIP_A3.txt

AMIP A3 1850allF/home/t/tang30/2018/E3SM_simulations/20180716.DECKv1b_A3_1850allF.ne30_oEC.edison5 TB

No failures out of 19 tar files.

AMIP_A3_1850allF.txtAMIP A3 1850aeroF/home/t/tang30/2018/E3SM_simulations/20180716.DECKv1b_A3_1850aeroF.ne30_oEC.edison5 TBNo failures out of 19 tar files.AMIP_A3_1850aeroF.txt
Total
161 TB18/634 = 2.839% tar files failed.

Known bad data

Some users have reported errors when trying to extract from HPSS archive. The cause of the bad data is currently unknown; possibilities include a bug in zstash or data file being corrupted to/on HPSS.

...

  • Could extract files; they look like netCDF files but md5 checksums do not match original ones.

...

  • Could extract first file, but nothing else.

...

  • Chris Golaz was able to successfully rescue contents of tar archive; md5 checksums matched original ones.
  • Data available under:

/global/cscratch1/sd/golaz/ACME_simulations/rescue/20180302.DECKv1b_H3.ne30_oEC.edison

Retrieving data with zstash

...

Code Block
languagebash
zstash extract --hpss=/home/g/golaz/2018/E3SM_simulations/repaired/20180129.DECKv1b_piControl.ne30_oEC.edison "*.cam.h0.00[3-6]?-??.nc"

where /home/g/golaz/2018/E3SM_simulations/repaired/20180129.DECKv1b_piControl.ne30_oEC.edison is the path on HPSS listed in the table above, and "*.cam.h0.00[3-6]?-??.nc" is a matching pattern to select all cam.h0 files between years 0030 and 0069, enclosing it in double quotes avoids shell level substitution.

...