[This is a draft!]
This document is to provide guidelines for managing our E3SM simulations that need to be archived, including official simulations and important internal working simulations. The following steps are required to be done immediately after simulation completion to avoid data loss/corruption and time consuming re-runs.
1. Short term archiving
(Optional, but recommended). Use CIME short-term archiving utility to reorganize model output to avoid having tens of thousands of output files in the single run/ sub-directory. Typical usage:
cd case_scripts ./case.st_archive --force-move
The ‘force-move’ option will greatly speed-up short term archiving by moving rather than copying files. You can also invoke short term archiving periodically while the simulation is progressing. To do so safely, use additional options:
cd case_scripts ./case.st_archive --force-move --last-date 2005-01-01 --no-incomplete-logs
2. Archive to NERSC HPSS using zstash
The original model output should be archived on NERSC HPSS using zstash:
If a simulation was ran on systems with HPSS: run zstash to archive the data to the local HPSS tape. If the HPSS is not on NERSC, use Globus to transfer the zstash *.tar and index.db files to NERSC HPSS:
If data were archived on Theta HPSS, to transfer to NERSC HPSS, use the Globus endpoint alcf#dtn_hpss and NERSC HPSS. (Make sure to opt for “verify file integrity after transfer“ as a transfer and Sync option, which checks if source and destination file checksums match)
If a simulation was ran on a system without HPSS (i.e., anvil and compy): run zstash to generate index.db and tar files locally [Note: this zstash feature is under development will be available soon, ~within this month?] and transfer these files to NERSC HPSS using Globus.
Notes on using zstash:
To archive output from an E3SM simulation located under $CSCRATCH/E3SM_simulations/20170731.F20TR.ne30_ne30.edison:
$ cd $CSCRATCH/E3SM_simulations/20170731.F20TR.ne30_ne30.edison $ zstash create --hpss=test/E3SM_simulations/20170731.F20TR.ne30_ne30.edison .
The above command should generate optimized tar file(s) and the corresponding index database (index.db), saving checksums and additional metadata of the tar file(s).
It is highly recommended that you verify the integrity of the tar files after archiving with zstash on both the local machine and NERSC. The safest way to do so is to go to a new, empty directory and run:
$ zstash check --hpss=test/E3SM_simulations/20170731.F20TR.ne30_ne30.edison
If you encounter an error, save your original data. You may need to re-upload it via zstash create
The zstash documentation and best practice for E3SM can be found here.
3. Document HPSS location
Document the HPSS locations on confluence so that members from infrastructure group can be notified and move the data to a centralized space at LLNL for further post-processing (i.e. ESGF publication). Each simulation group has its own confluence pages documenting the simulation output archive location:
Cryosphere[coming soon]
Note: For non-official simulations (one-off tuning runs, sensitivity study for research papers, etc.), please have a separate table with a note column, indicating how long the archive is desired to be kept or “don’t delete until data being published to ESGF“. These files may be given lower priority to be kept once our NERSC HPSS quota is reached.