The purpose of this document page is to provide guidelines document E3SM project’s policy for managing our the E3SM simulations that need to be archived, including official simulations and important internal working simulations. The following steps are required to be done immediately after simulation completion to avoid data loss/corruption and time consuming re-runs.
Note E3SM’s policy change: All 3 steps are now required steps.
1. Short term archiving
(Optional, but recommended). Please note that this is a required step per latest E3SM policy.
Use CIME short-term archiving utility to reorganize model output to avoid having tens of thousands of output files in the single run/ sub-directory. Typical usage:
...
2. Archive to NERSC HPSS using zstash
Note that using zstash is required when archiving E3SM data. For the systems that do not provide HPSS, use zsash with “--hpss=none” to create a tar files to be then copied to LLNL for permanent storage.
The original model output should be archived on NERSC HPSS using zstash:
...
Documenting the HPSS locations on a central confluence page is a required step and it is helpful for everyone in the project who might have a need to locate the files. Members from infrastructure group will be closely monitoring these pages. Once a new simulation is entered, the data will be copied to a centralized space at LLNL (Long-Term E3SM Archive at LLNL - Data Source and Transfer Status and Documentation ) for further post-processing (i.e. ESGF publication). A default set of simulation data will be published to ESGF for official simulations. Please drop an email to Jill Chengzhu Zhang (zhang40@llnl.gov) and Sterling Baldwin (Unlicensed) (baldwin32@llnl.gov) for special publication requirements. Each simulation and NGD group has its own confluence pages documenting the simulation output archive locations:
...