Contact: Ryan Forsyth
Long Term Archiving with zstash
Info |
---|
Summary
|
...
Edit the file and customize as needed. The file is structured with [section]
and [[sub-sections]]
. There is a [default]
section, followed by additional sections for each available zppy
task (climo, ts, e3sm_diags, mpas_analysis, …). Sub-sections can be used to have multiple instances of a particular task, for example regridded monthly or globally averaged time series files. Refer to the zppy documentation for more details.
...
Code Block |
---|
cd <simulations_dir>/<case_name>/post/scripts cat *.status # should be a list of "OK" grep -v "OK" *.status # lists files without "OK" |
If you re-run zppy
, it will check status of tasks and will skip any task if its status is “OK”.
As your simulation progresses, you can update the post-processing years in the configuration file and re-run zppy
. Newly added task will be submitted, while previously completed ones will be skipped.
...
Make a bulleted list of links, e.g., for https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/<username>/E3SM/v2/beta/<case_name>/mpas_analysis/ts_0001-0050_climo_0021-0050/
, create a bullet “1-50 (time series), 21-50 (climatology)”.
Long Term Archiving with zstash
Simulations that are deemed sufficiently valuable should be archived using zstash
for long-term preservation.
...
Compy, Anvil and Chrysalis do not have local HPSS. We rely on NERSC HPSS for long-term archiving. Archiving requires a few separate steps:
Run '
zstash create
' to archive to local disk.Using Globus, transfer
zstash
archive files (everything under thezstash/
subdirectory) to NERSC HPSS. Select the option to preserve original files modification date.Run '
zstash check
' to verify integrity ofzstash
archive (and their transfer).Update simulation Confluence page with path to HPSS.
...