The purpose of this page is to document E3SM project’s policy for managing the E3SM simulations that need to be archived, including official simulations and important internal working simulations. The following steps are required to be done immediately after simulation completion to avoid data loss/corruption and time consuming re-runs.
Note E3SM’s policy change: All 3 steps are now required steps.
1. Short term archiving
...
The zstash documentation and best practices for E3SM can be found here.at: https://e3sm-project.github.io/zstash/_build/html/master/index.html
3. Document HPSS locations
Documenting the HPSS locations on a central confluence page is a required step and it is helpful for everyone in the project who might have a need to locate the files. Members from infrastructure group will be closely monitoring these pages. Once a new simulation is entered, the data will be copied to a centralized space at LLNL (E3SM Archive - Data Source and Transfer Status ) for further post-processing (i.e. ESGF publication). A default set of simulation data will be published to ESGF for official simulations. Please drop an email to Jill Chengzhu Zhang (zhang40@llnl.gov) and Sterling Baldwin (Unlicensed) (baldwin32@llnle3sm-data-support@llnl.gov ) for special publication requirements . Each simulation (i.e., specify years, cmip publication, etc.).
For v2: a project wide page is created /wiki/spaces/ED/pages/2766340117, and all the production runs are required to have an additional copy available through the NERSC Science Gateway:
Path on NERSC HPSS: /home/projects/e3sm/www
URL on the Web: https://portal.nersc.gov/archive/home/projects/e3sm/www/
For v1: Simulation and NGD group has its own confluence pages documenting the simulation output archive locations::
Water Cycle: DECK v1, High-res Coupled v1
BGC: BGC v1
Cryosphere: Cryosphere v1
NGD-Atmosphere Physics
NGD-Land
NGD-SCREAM
Note: For non-official simulations (one-off tuning runs, sensitivity study for research papers, etc.), please have a separate table with a note column, indicating how long the archive is desired to be kept or “don’t delete until data being published to ESGF“. These files may be given lower priority to be kept once our NERSC HPSS quota is reached. /wiki/spaces/EBGC/pages/1056145509
Cryosphere: /wiki/spaces/ECG/pages/1736933506