The purpose of this page is to document E3SM project’s policy for managing the E3SM simulations that need to be archived, including official simulations and important internal working simulations. The following steps are required to be done immediately after simulation completion to avoid data loss/corruption and time consuming re-runs.
Note E3SM’s policy change: All 3 steps are now required steps.
1. Short term archiving
...
If a simulation was run on systems with HPSS: run zstash to archive the data to the local HPSS tape. If the HPSS is not on NERSC, use Globus to transfer the zstash *.tar and index.db files to NERSC HPSS:
If data were archived on Theta HPSS, to transfer to NERSC HPSS, use the Globus endpoint alcf#dtn_hpss and NERSC HPSS. (Make sure to opt for “verify file integrity after transfer“ as a transfer and Sync option, which checks if source and destination file checksums match)
If a simulation was run on a system without HPSS (i.e., anvil and compy): run zstash to generate index.db and tar files locally [Note: this zstash feature is available after version 0.4.1] and transfer these files to NERSC HPSS using Globus].
Make sure permissions are correct for files on HPSS (Please make the directory group/project readable, once publication is authorized that will be changed to world-readable).
If you keep seeing issues with a specific transfer between endpoints, you can email
support@globus.orgwith the transfer ID. They can look at the transfer logs and facilitate the conversation between the endpoint admins.
...