It is resolved that E3SM will maintain an LLNL-local copy of all production E3SM simulation run archives, and that future publication (and pre-publication) activities will proceed from these local, zstash-format archives. Specifically,
The LLNL E3SM Archives will retain the state of simulation output as released by the corresponding project groups (transferred “as is” from NERSC HPSS). Certain “mapping functions” will be conducted to identify the publishable content within the (often variably-structured) archives in order to facilitate survey and extraction of individual native-output datasets.
Data not already published will be prepared for publication by being extracted to a staging area, whereupon occasional data irregularities (missing data, overlapping data from unusual restarts) will be addressed, and default post processing (regridding for selected time-series, climatology generation) will proceed. The data will reside in staging under a pre-publication “warehouse” directory, with faceted subdirectory locations exactly matching the eventual publication directory hierarchies. Where data is not yet requested for publication, it will reside indefinitely in the pre-publication warehouse.
Where data is thereafter (or already) scheduled for publication, it need only be moved (relinked) to the corresponding publication directory path, mapfile generated, and the formal ESGF publication process engaged.
By this process, almost all queries regarding E3SM data production runs can be answered by tools that apply themselves (transparently if desired) to either their pre-publication or publication location, largely obviating the continued dependence upon the “archive mapping functions” except where very archive-specific questions need to be addressed. The data footprint does not increase, since datasets pulled from archive will reside in exactly one of the two locations.
LLNL E3SM Archives
Location: /p/user_pub/e3sm/archives/<model>/<campaign>/<archive_directory>/
Guides and Tools: . . .
LLNL E3SM Staging “Warehouse”
Location: /p/user_pub/e3sm/staging/prepub/<faceted_dataset_directory>/
Guides and Tools: . . .
LLNL E3SM Publication
Location: /p/user_pub/work/E3SM/<faceted_dataset_directory>/
Guides and Tools: . . .