Versions Compared

Old Version 14

changes.mady.by.user Chris Golaz

Saved on

compared with

New Version Current

changes.mady.by.user Qi Tang

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Warning
iconfalse

  NERSC Directory Change Notice  

Due to project's name change at NERSC from 'ACME' to 'E3SM' and NERSC’s file system update, the directory  '/project/projectdirs/acme/' is now '/cfs/cdirs/e3sm'

NERSC

DECK v1 simulation original model output has been archived on NERSC HPSS using zstash (the quickstart quide is https://e3sm-project.github.io/zstash/docs/html/quickstart.html). Please contact Zeshawn Shaheen (Unlicensed) for help with zstash. See below for some examples on how to retrieve files using zstash. For a more information, refer to the zstash documentation. The table below lists output location for the various simulations. Note that some archives initially contained a few corrupted files. These problems have all been corrected.

SimulationHPSS locationApproximate size
Failed files and tarballs
zstash check output
piControl

/home/g/golaz/2018/E3SM_simulations/repaired/20180129.DECKv1b_piControl.ne30_oEC.edison

37 TBStill checking.Still checking.abrupt-4xCO2

Alternate location:
/home/t/tang30/2018/E3SM_simulations/repaired/20180129.DECKv1b_piControl.ne30_oEC.edison

37 TB
abrupt-4xCO2

/home/g/golaz/2018/E3SM_simulations/repaired/20180215.DECKv1b_abrupt4xCO2.ne30_oEC.edison
Alternate location:
/home/t/tang30/2018/E3SM_simulations/repaired/20180215.DECKv1b_abrupt4xCO2.ne30_oEC.edison

12 TB

abrupt-4xCO2

-failed.txt
00002a.tar
00001d.tar
00000f.tar
000028.tar
000021.tarTotal of 46 tar files

abrupt-4xCO2.txt

Historical H1

(extension to 300 years)

/home/g/golaz/2019/E3SM_simulations/20190722.DECKv1b_abrupt4xCO2.ne30_oEC3.compy
Alternate location:
/home/t/tang30/2019/E3SM_simulations/20190722.DECKv1b_abrupt4xCO2.ne30_oEC3.compy
13 TB
1pctCO2

/home/g/golaz/2018/E3SM_simulations/repaired/20180215.DECKv1b_1pctCO2.ne30_oEC.edison

11 TB

1pctCO2-failed.txt
000000.tar

Total of 44 tar files

1pctCO2.txt

Historical H2

Alternate location:
/home/t/tang30/2018/E3SM_simulations/repaired/20180215.DECKv1b_1pctCO2.ne30_oEC.edison

11 TB
Historical H1

/home/g/golaz/2018/E3SM_simulations/repaired/20180215.DECKv1b_H1.ne30_oEC.edison
Alternate location:
/home/t/tang30/2018/E3SM_simulations/repaired/20180215.DECKv1b_H1.ne30_oEC.edison

12 TB
Still checking.Still checking.
Historical H2

/home/g/golaz/2018/E3SM_simulations/repaired/20180220.DECKv1b_H2.ne30_oEC.edison
Alternate location:
/home/t/tang30/2018/E3SM_simulations/repaired/20180220.DECKv1b_H2.ne30_oEC.edison

12 TB
Historical
_H2-failed.txt
000014.tar
00002d.tarTotal of 49 tar filesHistorical_H2.txtHistorical H3
H3

/home/z/zshaheen/2018/E3SM_simulations/repaired/20180302.DECKv1b_H3.ne30_oEC.edison

Alternate locations:

/home/g/golaz/2018/E3SM_simulations/repaired/20180302.DECKv1b_H3.ne30_oEC.edison

12 TB
Historical
_H3-failed.txt
000000.tar
00002a.tar
00001f.tar
000025.tarTotal of 49 tar files

Historical_H3.txt

Historical H4
H4

/home/t/tang30/2018/E3SM_simulations/20180305.DECKv1b_H4.ne30_oEC.edison
Alternate location:
/home/

t

g/

tang30

golaz/2018/E3SM_simulations/20180305.DECKv1b_H4.ne30_oEC.edison

12 TB

No failures.

Historical
_H4.txtHistorical Historical_H5.txt
H5/home/g/golaz/2018/E3SM_simulations/20180307.DECKv1b_H5.ne30_oEC.edison
12 TBNo failures.AMIP_A1.txt
Alternate location:
/home/t/tang30/2018/E3SM_simulations/20180307.DECKv1b_H5.ne30_oEC.edison		12 TB
AMIP A1

/home/t/tang30/2018/E3SM_simulations/20180316.DECKv1b_A1.ne30_oEC.edison

4 TBNo failures.AMIP

Alternate location:

/home/g/golaz/2018/E3SM_simulations/20180316.DECKv1b_A1.ne30_oEC.edison

4 TB
AMIP A1 1850allF

/home/t/tang30/2018/E3SM_simulations/20180508.DECKv1b_A1_1850allF.ne30_oEC.edison

5 TBNo failures.

Alternate location:

/home/g/golaz/2018/E3SM_simulations/20180508.DECKv1b_A1_1850allF

.txtAMIP

.ne30_oEC.edison

5 TB
AMIP A1 1850aeroF

/home/t/tang30/2018/E3SM_simulations/20180508.DECKv1b_A1_1850aeroF.ne30_oEC.edison

5 TBNo failures.

Alternate location:

/home/g/golaz/2018/E3SM_simulations/20180508.DECKv1b_A1_1850aeroF

.txtAMIP_A2.txt

.ne30_oEC.edison

5 TB
AMIP A2

/home/t/tang30/2018/E3SM_simulations/20180622.DECKv1b_A2.ne30_oEC.edison

4 TBNo failures.AMIP

Alternate location:

/home/g/golaz/2018/E3SM_simulations/20180622.DECKv1b_A2.ne30_oEC.edison

4 TB
AMIP A2 1850allF

/home/t/tang30/2018/E3SM_simulations/20180622.DECKv1b_A2_1850allF.ne30_oEC.edison

4 TBNo failures.

Alternate location:

/home/g/golaz/2018/E3SM_simulations/20180622.DECKv1b_A2_1850allF.

txtAMIP

ne30_oEC.edison

4 TB
AMIP A2 1850aeroF

/home/t/tang30/2018/E3SM_simulations/20180622.DECKv1b_A2_1850aeroF.ne30_oEC.edison

4 TBNo failures.

Alternate location:

/home/g/golaz/2018/E3SM_simulations/20180622.DECKv1b_A2_1850aeroF

.txtAMIP_A3.txt

.ne30_oEC.edison

4 TB
AMIP A3

/home/t/tang30/2018/E3SM_simulations/20180716.DECKv1b_A3.ne30_oEC.edison

5 TBNo failures.AMIP

Alternate location:

/home/g/golaz/2018/E3SM_simulations/20180716.DECKv1b_A3.ne30_oEC.edison

5 TB
AMIP A3 1850allF

/home/t/tang30/2018/E3SM_simulations/20180716.DECKv1b_A3_1850allF.ne30_oEC.edison

5 TB

No failures.

Alternate location:

/home/g/golaz/2018/E3SM_simulations/20180716.DECKv1b_A3_1850allF

.txt

.ne30_oEC.edison

5 TB
AMIP A3 1850aeroF

/home/t/tang30/2018/E3SM_simulations/20180716.DECKv1b_A3_1850aeroF.ne30_oEC.edison

Alternate location:

/home/g/golaz/2018/E3SM_simulations/20180716.DECKv1b_A3_1850aeroF.ne30_oEC.edison

5 TB

No failures.

AMIP_A3_1850aeroF.txtTotal161 TB

Known bad data

Some users have reported errors when trying to extract from HPSS archive. The cause of the bad data is currently unknown; possibilities include a bug in zstash or data file being corrupted to/on HPSS.

SimulationTar archiveContentRescue statusabrupt-4xCO200002a.tarabrupt4xCO2_index00002a.txt
  • Could extract files; they look like netCDF files but md5 checksums do not match original ones.
Historical H200002d.tarH2_index00002d.txt
  • Could extract first file, but nothing else.
Historical H3000000.tarH3_index000000.txt
  • Chris Golaz was able to successfully rescue contents of tar archive; md5 checksums matched original ones.
  • Data available under:
/global/cscratch1/sd/golaz/ACME_simulations/rescue/20180302.DECKv1b_H3.ne30_oEC.edison
DAMIP hist-GHG H1

/home/t/tang30/2019/E3SM_simulations/20190404.DECKv1b_H1_hist-GHG.ne30_oEC.edison

Alternate location:

/home/g/golaz/2019/E3SM_simulations/20190404.DECKv1b_H1_hist-GHG.ne30_oEC.edison

12 TB
DAMIP hist-GHG H2

/home/t/tang30/2019/E3SM_simulations/20190404.DECKv1b_H2_hist-GHG.ne30_oEC.edison

Alternate location:

/home/g/golaz/2019/E3SM_simulations/20190404.DECKv1b_H2_hist-GHG.ne30_oEC.edison

12 TB
DAMIP hist-GHG H3

/home/t/tang30/2019/E3SM_simulations/20190404.DECKv1b_H3_hist-GHG.ne30_oEC.edison

Alternate location:

/home/g/golaz/2018/E3SM_simulations/repaired/20180302.DECKv1b_H3.ne30_oEC.edison

12 TB
Projection P1

/home/t/tang30/2019/E3SM_simulations/20190807.DECKv1b_P1_SSP5-8.5.ne30_oEC.cori-knl

Alternate location:

/home/g/golaz/2019/E3SM_simulations/20190807.DECKv1b_P1_SSP5-8.5.ne30_oEC.cori-knl

8 TB
Projection P2

/home/t/tang30/2019/E3SM_simulations/20190821.DECKv1b_P2_SSP5-8.5.ne30_oEC.cori-knl

Alternate location:

/home/g/golaz/2019/E3SM_simulations/20190821.DECKv1b_P2_SSP5-8.5.ne30_oEC.cori-knl

8 TB
Projection P3

/home/t/tang30/2019/E3SM_simulations/20190821.DECKv1b_P3_SSP5-8.5.ne30_oEC.cori-knl

Alternate location:

/home/g/golaz/2019/E3SM_simulations/20190821.DECKv1b_P3_SSP5-8.5.ne30_oEC.cori-knl

8 TB
Projection P4

/home/t/tang30/2019/E3SM_simulations/20190821.DECKv1b_P4_SSP5-8.5.ne30_oEC.cori-knl

Alternate location:

/home/g/golaz/2019/E3SM_simulations/20190821.DECKv1b_P4_SSP5-8.5.ne30_oEC.cori-knl

8 TB
Projection P5

/home/t/tang30/2019/E3SM_simulations/20190821.DECKv1b_P5_SSP5-8.5.ne30_oEC.cori-knl

Alternate location:

/home/g/golaz/2019/E3SM_simulations/20190821.DECKv1b_P5_SSP5-8.5.ne30_oEC.cori-knl

8 TB
DAMIP SSP5-8.5-GHG P1

/home/t/tang30/2019/E3SM_simulations/20191019.DECKv1b_P1_SSP5-8.5-GHG.ne30_oEC.cori-knl

Alternate location:

/home/g/golaz/2019/E3SM_simulations/20191019.DECKv1b_P1_SSP5-8.5-GHG.ne30_oEC.cori-knl

8 TB
DAMIP SSP5-8.5-GHG P2

/home/t/tang30/2019/E3SM_simulations/20191019.DECKv1b_P2_SSP5-8.5-GHG.ne30_oEC.cori-knl

Alternate location:

/home/g/golaz/2019/E3SM_simulations/20191019.DECKv1b_P2_SSP5-8.5-GHG.ne30_oEC.cori-knl

8 TB
DAMIP SSP5-8.5-GHG P3

/home/t/tang30/2019/E3SM_simulations/20191019.DECKv1b_P3_SSP5-8.5-GHG.ne30_oEC.cori-knl

Alternate location:

/home/g/golaz/2019/E3SM_simulations/20191019.DECKv1b_P3_SSP5-8.5-GHG.ne30_oEC.cori-knl

8 TB
Total
261 TB

Retrieving data with zstash

Below are some basic examples on how to retrieve DECK output data on NERSC. You can retrieve directly from NERSC Cori machine, but for better performance we strongly recommend login to one of the NERSC DTNs (data transfer nodes: dtn<01…04>.nersc.gov). The example below relies on the E3SM unified environment. Alternatively, you can also install zstash in your own conda environment (see zstash documentation).

Activate environment

If you are using bash shell:

Code Block
languagebash
source /global/cfs/cdirs/e3sm/software/anaconda_envs/load_latest_e3sm_unified.sh

If you are using csh shell:

Code Block
languagebash
source /global/cfs/cdirs/e3sm/software/anaconda_envs/load_latest_e3sm_unified.csh

Examples

Retrieve selected cam.h0 files from piControl

Let's say you want to retrieve all cam.h0 (monthly atmosphere output files) between years 0030 and 0069 for the piControl simulation. The zstash command would be

Code Block
languagebash
zstash extract --hpss=/home/g/golaz/2018/E3SM_simulations/repaired/20180129.DECKv1b_piControl.ne30_oEC.edison "*.cam.h0.00[3-6]?-??.nc"

where /home/g/golaz/2018/E3SM_simulations/repaired/20180129.DECKv1b_piControl.ne30_oEC.edison is the path on HPSS listed in the table above, and "*.cam.h0.00[3-6]?-??.nc" is a matching pattern to select all cam.h0 files between years 0030 and 0069, enclosing it in double quotes avoids shell level substitution.

The extracted files will be stored in a sub-directory structure that preserves the original structure:

Code Block
languagebash
archive/atm/hist/20180129.DECKv1b_piControl.ne30_oEC.edison.cam.h0.0030-01.nc                              
archive/atm/hist/20180129.DECKv1b_piControl.ne30_oEC.edison.cam.h0.0030-02.nc                              
archive/atm/hist/20180129.DECKv1b_piControl.ne30_oEC.edison.cam.h0.0030-03.nc                              
archive/atm/hist/20180129.DECKv1b_piControl.ne30_oEC.edison.cam.h0.0030-04.nc

[...]

archive/atm/hist/20180129.DECKv1b_piControl.ne30_oEC.edison.cam.h0.0069-09.nc
archive/atm/hist/20180129.DECKv1b_piControl.ne30_oEC.edison.cam.h0.0069-10.nc
archive/atm/hist/20180129.DECKv1b_piControl.ne30_oEC.edison.cam.h0.0069-11.nc
archive/atm/hist/20180129.DECKv1b_piControl.ne30_oEC.edison.cam.h0.0069-12.nc