...
This document is to provide guidelines for managing our E3SM simulations that need to be archived, including official simulations and important internal working simulations. The following steps are required to be done immediately after simulation completion to avoid data loss/corruption and time consuming re-runs.
1. Short term archiving
(Optional, but recommended). Use CIME short-term archiving utility to reorganize model output to avoid having tens of thousands of output files in the single run/ sub-directory. Typical usage:
| Code Block |
|---|
cd case_scripts ./case.st_archive --force-move |
...
| Code Block |
|---|
cd case_scripts ./case.st_archive --force-move --last-date 2005-01-01 --no-incomplete-logs |
2. Archive to NERSC HPSS using zstash
The original model output should be archived on NERSC HPSS using zstash:
...
Note: For non-official simulations (one-off, tuning runs), please attach an expiration date to the name of the archive folder. These files will be given lower priority to be kept once our NERSC HPSS quota is reached.
3. Document HPSS location
Document the HPSS locations on confluence so that members from infrastructure group can be notified and move the data to a centralized space at LLNL for further post-processing (i.e. ESGF publication). Each simulation group has its own confluence pages documenting the simulation output archive location:
...