...
Use ncclimo if possible. It requires and comes with NCO version 4.6.0 and later. Since about early 2018, the preferred way to obtain NCO for E3SM analysis is with the E3SM-Unified Conda package, which installs numerous analysis packages in a platform-independent manner and, as importantly, allows you to skip reading the rest of this paragraph. Those who need only NCO, or who wish to avoid Conda, should read-on. The newest versions of NCO are installed on rhea.ccs.ornl.gov at ORNL, blues/anvil.lcrc.anl.gov: at ANL, cori.nersc.gov (NERSC), aims4.llnl.gov (LLNL), cheyenne.ucar.edu (NCAR), and compy.pnl.gov (PNNL). The ncclimo and ncremap scripts should find the latest versions automatically, and not require any module or path changes. To use other (besides the ncclimo and ncremap scripts) NCO executables from the command-line or from your own scripts may require loading modules. This is site-specific and not under my (CZ's) control. You might check that the default NCO is recent enough (try module load nco, then ncks --version) or use developers' executables/libraries (in ~zender/[bin,lib] on all machines). Follow these directions on the NCO homepage to install on your own machines/directories. It can be as easy as apt-get install nco, dnf install nco, or conda install -c conda-forge nco, or you can build/install from scratch with configure;make;make install.
Timeseries Reshaping mode, aka Splitting
...
ncclimo will reshape a series of input files into outputs that are continuous timeseries of each variable taken from all input files. Timeseries to be reshaped (split) often come with hard-to-predict names, e.g., because the number of days or months in a file, or timesteps per day or month may all vary. Thus ncclimo in splitter mode requires the user to supply the input filenames. ncclimo will not construct input filenames itself in splitter mode (unlike monthly or annual climo generation mode). ncclimo will, as of version 4.6.4, automatically switch to timeseries reshaping mode if it receives a list of files through a pipe to stdin, or, alternatively, placed as positional arguments (after the last command-line option), or if neither of these is done and no caseid is specified, in which case it assumes all *.nc files in drc_in constitute the input file list. These examples invoke reshaping mode in the four possible ways (choose your poison):
...
The output is a collection of per-variable timeseries such as FSNT_YYYYMM_YYYYMM.nc, AODVIS_YYYYMM_YYYYMM.nc, etc. The output is split into segments each containing no more than ypf_max (default 50) years-per-file, e.g., FSNT_000101_005012.nc, FSNT_005101_009912.nc, FSNT_010001_014912.nc, etc. Change the maximum number of years-per-output-file with the --ypf_max=ypf_max option. One caveat worth noting is that, for technical reasons, non-interactive batch queues cannot unambiguously distinguish filelists provided via stdin from batch queue information provided via stdin. To address this issue, ncclimo supports the --stdin switch to cause the filelist to be read from stdin. This switch is only necessary when running ncclimo in non-interactive batch mode. The switch is permitted though redundant when running ncclimo interactively (i.e., from a terminal shell). The remainder of the ncclimo documentation refers to climatology-generation mode, not to splitter mode.
Climatology generation mode (produce monthly + seasonal + annual climos from monthly input files)
...
The usual way to use ncclimo is to bring up a terminal window and type:
...
When invoked without options ncclimo outputs a handy table of all available options, their long-option synonyms, and some examples. NCO documentation here describes the full meaning of all options. A short summary of the most common options is:
-a, --dec_md: The “December mode” specifies the desired type of DJF average. Either -a sdd (default) or -a scd. scd means seasonally continuous December. The first month used will be Dec of the year before the start year you specify with -s. sdd means seasonally discontinuous December. The first month used will be Jan of the specified start year. (Prior to NCO 4.9.4, released in August 2020, the default was scd not sdd.)
-C, --clm_md: Climatology mode. Either mth (default, and for monthly-mean input files), hfc for high-frequency-climate (i.e., sub-daily) mean input files, or ann (for annual-mean input files).
-c: caseid, i.e., , --caseid: The case ID or simulation name. For input files like famipc5_ne30_v0.3_00001.cam.h0.1980-01.nc, specify -c famipc5_ne30_v0.3_00001. The .cam. and .h0. bits are added to the filenames internally by default, and can be modified via the -m mdl_nm and -h hst_nm switches if needed. See comments in ncclimo for documentation.
-e: end , --yr_end: End year (example: 2000). Unless the optional flag -a scdis specified, the last month used will be Dec of the specified end year. If -a scd is specified, the last month will be Nov of the specified end year.
-h, --hst_nm: history History file volume that separates the model name from the date in the input file name. Default is h0. Other common values are h1 and h.
-i: directory , --drc_in: Directory containing all netCDF files to be used for input to this code.
-m: model type, --mdl_nm: Model name. Default is cam. Other options are clm2, ocn cism, ice cice, cism mpasseaice, cice mpaso, pop.
-o: directory , --drc_out: Directory where computed native grid climo files will be placed. Regridded climos will also be placed here unless a separate directory for them is specified with -O (NB: capital "O")
-O, --drc_rgr: directory where regridded climo files will be placed.
-s, --yr_srt: start year (example: 1980). The By default, the first month used will be Jan of the year the this start year you specify. If -a scd is specified, the first month used will be Dec of the year before this specified start year (example Dec 1979 to allow for contiguous DJF climos).
-v, --var: variable list, e.g., FSNT,AODVIS,PREC.? (yes, regular expressions work so this expands to PRECC,PRECL,PRECSC,PRECSL)
MPAS
...
considerations
...
MPAS ocean and ice models currently have their own (non-CESM'ish) naming convention that guarantees output files have the same names for all simulations. By default ncclimo analyzes the "timeSeriesStatsMonthly" analysis member output (tell CZ if you want options for other AM output). ncclimo recognizes input files as being MPAS-style when invoked with "-m mpaso" or "-m mpascice" (or synonyms) like this:
| Code Block |
|---|
ncclimo -m mpaso -s 1980 -e 1983 -i drc_in -o drc_out # MPAS-O |
| Code Block |
ncclimo -m mpascicempasseaice -s 1980 -e 1983 -i drc_in -o drc_out # MPAS-I |
...
The climatological annual mean, ANN, is also computed automatically when MAM,JJA,SON, and DJF are all requested (which is the default, so ANN is always computed by default). Use "–seasons–-seasons=none" to completely turn-off seasonal and annual-mean climatologies.
MPAS climos are unaware of missing values until/unless the input files are "fixed". I We recommend that the person who produces the simulation annotate all floating point variables with the appropriate _FillValue prior to invoking ncclimo. Run something like this once in the history file directory:
| Code Block |
|---|
for fl in `ls hist.*` ; do
ncatted -O -t -a _FillValue,,o,d,-9.99999979021476795361e+33 ${fl}
done |
If/when MPAS O/I generates the _FillValue attributes itself, this step can and should be skipped. All other ncclimo features like regridding (below) are invoked identically for MPAS as for EAM/ELM users although under-the-hood ncclimo does do some special pre-processing (dimension permutation, metadata annotation) for MPAS. A five-year oEC60to30 MPAS-O climo with regridding to T62 takes < 10 minutes on rhea.
Annual climos (produce
...
climos from annual-mean input files)
...
Not all model or observed history files are created as monthly means. To create a climatological annual mean from a series of annual mean inputs (such as from a land ice model), select ncclimo's annual climatology mode with the "-C ann" option:
| Code Block |
|---|
ncclimo -C ann -m cism -h h -c caseid -s 1851 -e 1900 -i drc_in -o drc_out |
The options "-m mdl_nm" and "-h hst_nm" (that default to "cam" and "h0", respectively) tell ncclimo how to construct the input filenames. The above formula names the files caseid.cism.h.1851-01-01-00000.nc, caseid.cism.h.1852-01-01-00000.nc, and so on. Annual climatology mode produces a single output file (or two if regridding is selected), and in all other respects behaves the same as monthly climatology mode.
Daily climos (interannual day-of-year statistics from multi-year daily-to-diurnal-mean input files)
...
High frequency timeseries are often available as daily means. To create a climatological daily mean from a series of daily mean inputs, select daily mode with the "--clm_md=dly" option. What is computed? In daily mode ncclimo produces 365 output files, each consisting of the interannual average of the given day-of-year. If the input data are at sub-daily resolution (e.g., 8 timesteps per day for three hourly data), then by default all timesteps in a day are averaged into the daily output. An option to preserve the diurnal cycle for each day is in the works, contact Charlie Zender if interested. Also on the drawing board is a simple method to obtain min/max/avg etc. at the same time, or at least with the same command.
...