...
If a simulation was run on systems with HPSS: run zstash to archive the data to the local HPSS tape. If the HPSS is not on NERSC, use Globus to transfer the zstash *.tar and index.db files to NERSC HPSS:
If data were archived on Theta HPSS, to transfer to NERSC HPSS, use the Globus endpoint alcf#dtn_hpss and NERSC HPSS. (Make sure to opt for “verify file integrity after transfer“ as a transfer and Sync option, which checks if source and destination file checksums match)
If a simulation was run on a system without HPSS (i.e., anvil and compy): run zstash to generate index.db and tar files locally [Note: this zstash feature is under development and will be available soon, ~within this month?] and transfer these files to NERSC HPSS using Globus.
Make sure permissions are correct for files on HPSS.
Notes on using zstash:
To archive output from an E3SM simulation located under $CSCRATCH/E3SM_simulations/20170731.F20TR.ne30_ne30.edison:
...
Documenting the HPSS locations on a central confluence page is helpful for everyone in the project who might have a need to locate the files. Members from infrastructure group will be closely monitoring these pages. Once a new simulation is entered, the data will be copied to a centralized space at LLNL for further post-processing (i.e. ESGF publication).[Make sure permissions are correct for files on hpss.] A default set of simulation data will be published to ESGF for official simulations. Please drop an email to Jill Chengzhu Zhang (zhang40@llnl.gov) and Sterling Baldwin (Unlicensed) (baldwin32@llnl.gov) for special publication requirements. Each simulation and NGD group has its own confluence pages documenting the simulation output archive locations:
...