| Table of Contents |
|---|
Overview
During the ACME era (c. 2015-16), we conducted extensive evaluation of AMWG, UV-CDAT, and NCO codes for generating and regridding climatology files (see /wiki/spaces/ATM/pages/29164477), and determined that NCO provides the most correct answers, has the best metadata, and is fastest. (These conclusions did not consider any Xarray or cloud-based methodologies, and so perhaps should be revisited.) The NCO climatology and regridding software ncclimo and ncremap, respectively, have evolved, and maintained back-compatibility so they now support all component models of E3SM (and CESM), up to and including EAMxx and MALI.
In climatology generation mode, the NCO operator ncclimo ingests "raw" data consisting of interannual sets of files, each containing sub-daily (diurnal), daily, monthly, or yearly averages, and from these produces climatological daily, monthly, seasonal, and/or annual means. Alternatively, in timeseries reshaping (aka “splitter”) mode, ncclimo will subset and temporally split the input timeseries into per-variable files spanning the entire period. ncclimo will optionally regrid (by calling ncremap) all output files in either mode. The primary ncremap documentation is here. This presentation, given at the Albuquerque workshop on 20151104, conveys much of the information presented below, and some newer information, in a more graphical format.
Prerequisites
ncclimo requires and comes with NCO version 4.6.0 and later. Since early 2018, the preferred way to obtain NCO for E3SM analysis is with the E3SM-Unified Conda package, which installs numerous analysis packages in a platform-independent manner and, as importantly, allows you to skip reading the rest of this paragraph. Those who need only NCO, or who wish to avoid Conda, should read-on. The newest versions of NCO are installed on all major DOE supercomputers in C. Zender’s home directory (usually ~zender/[bin,lib]), and semi-recent versions are sometimes available as machine modules (e.g., module load nco). This is site-specific and not under my (CZ's) control. Follow these directions on the NCO homepage to install on your own machines/directories. It can be as easy as apt-get install nco, dnf install nco, or conda install -c conda-forge nco, or you can build/install from scratch with configure;make;make install.
| Table of Contents |
|---|
Timeseries Reshaping mode, aka Splitting
ncclimo will reshape input files that are a series of snapshots of all model variables into outputs that are continuous timeseries of each individual variable taken from all input files. Timeseries to be reshaped (split) often come with hard-to-predict names, e.g., because the number of days or months in a file, or timesteps per day or month may all vary. Thus ncclimo in splitter mode requires the user to supply the input filenames. ncclimo will not construct input filenames itself in splitter mode (unlike monthly or annual climo generation mode). ncclimo will, as of version 5.0.4, employ timeseries reshaping mode if it receives the --split switch or the --ypf_max option described below. In addition, it must receive a list of files through a pipe to stdin, or, alternatively, placed as positional arguments (after the last command-line option), or if neither of these is done and no caseid is specified, in which case it assumes all *.nc files in drc_in constitute the input file list. These examples invoke reshaping mode in the four possible ways (choose your poison):
...