[This is a draft!]
This The purpose of this document is to provide guidelines for managing our E3SM simulations that need to be archived, including official simulations and important internal working simulations. The following steps are required to be done immediately after simulation completion to avoid data loss/corruption and time consuming re-runs.
...
The original model output should be archived on NERSC HPSS using zstash:
If a simulation was ran run on systems with HPSS: run zstash to archive the data to the local HPSS tape. If the HPSS is not on NERSC, use Globus to transfer the zstash *.tar and index.db files to NERSC HPSS:
If data were archived on Theta HPSS, to transfer to NERSC HPSS, use the Globus endpoint alcf#dtn_hpss and NERSC HPSS. (Make sure to opt for “verify file integrity after transfer“ as a transfer and Sync option, which checks if source and destination file checksums match)
If a simulation was ran run on a system without HPSS (i.e., anvil and compy): run zstash to generate index.db and tar files locally [Note: this zstash feature is under development and will be available soon, ~within this month?] and transfer these files to NERSC HPSS using Globus.
...
The zstash documentation and best practice practices for E3SM can be found here.
...