...
The LLNL E3SM Archives will retain the state of simulation output as released by the corresponding project groups (transferred “as is” from NERSC HPSS). Certain “mapping functions” will be conducted to identify the publishable content within the (often variably-structured) archives in order to facilitate survey and extraction of individual native-output datasets to the “faceted” pre-publication warehouse paths. The “maps” produced can be retained with the archives in the event that future access to the archives is required - however the goal is to avoid such need. Although initially kept on the local file system, as archives are “exhausted” (production materials warehoused or published) these archives can be pushed out to long-term tape storage, or eliminated, as policy dictates.
Data not already published[*] will be prepared for publication by being processed “in-situ” (state in “Assessment” or “Post-Processing”), whereupon occasional data irregularities (missing data, overlapping data from unusual restarts) will be addressed, and default post processing (regridding for selected time-series, climatology generation) will proceed. Upon completion of Assessment/Post-Processing the data will reside in the staging “warehouse” directory, with faceted subdirectory locations exactly matching the eventual publication directory hierarchies. Where data is not yet requested for publication, it will reside indefinitely in the pre-publication warehouse.
[*] Data already published, for which publication errors are discovered, will be treated as unpublished data (pulled from archive, cleaned, repaired) and the most effective update path to re-publication update will be engaged.)
Where new data is thereafter (or already) scheduled for publication, it need only be moved (relinked) to the corresponding publication directory path, mapfile generated, and the formal ESGF publication process engaged.
...