This page is devoted to instruction in ncremap. It describes steps necessary to create grids, and to regrid datasets between different grids with ncremap. Some of the simpler regridding options supported by ncclimo are also described at Generate, Regrid, and Split Climatologies (climo files) with ncclimo. This page describes those features in more detail, and other, more boutique features often useful for custom regridding solutions.
The Zen of Regridding
Most modern climate/weather-related research requires a regridding step in its workflow. The plethora of geometric and spectral grids on which model and observational data are stored ensures that regridding is usually necessary to scientific insight, especially the focused and variable resolution studies that E3SM models conduct. Why does such a common procedure seem so complex? Because a mind-boggling number of options are required to support advanced regridding features that many users never need. To defer that complexity, this HOWTO begins with solutions to the prototypical regridding problem, without mentioning any other options. It demonstrates how to solve that problem simply, including the minimal software installation required. Once the basic regridding vocabulary has been introduced, we solve the prototype problem when one or more inputs are "missing", or need to be created. The HOWTO ends with descriptions of different regridding modes and workflows that use features customized to particular models, observational datasets, and formats. The overall organization, including TBD sections (suggest others, or vote for prioritizing, below), is:
...
At a minimum, install a recent version NCO on your executable $PATH with the corresponding library on your $LD_LIBRARY_PATH. NCO installation instructions are here. Unless you have reason to do otherwise, we recommend installing NCO through the Conda package. The Conda NCO package automagically installs two other important regridding tools, the ESMF_RegridWeightGen (aka ERWG) executable and the TempestRemap (aka TR) executables. Execute 'ncremap --config' to verify you have a working installation:
...
This solution is deceptively simple because it conceals the choices, paths, and options required to create the appropriate map.nc for all situations. We will discuss creating map.nc later after showing more powerful and parallel ways to solve the prototype problem. For now, note that the The solution above only works for users savvy enough to know how to find appropriate pre-built map-files. E3SM v1.0 map-files are available here (fxm: link). At most DOE High Performance Computing (HPC) centers these . Many commonly used maps and grids can also be found in my (@czender's) directory directories as ~zender/data/[maps,grids] at most DOE High Performance Computing (HPC) centers. Take a minute now to look thereat these locations.
Pre-built map-files use the (nearly) standardized naming convention map_srcgrd_to_dstgrd_algtyp.YYYYMMDD.nc, where srcgrd and dstgrd are the source and destination grid names, algtyp is a shorthand for the numerical regridding algorithm, and YYYYMMDD is the date the map was created. The source grid in the example above is called ne30np4, the destination is called fv129x256. The only pre-built map for this combination is map.nc = map_ne30np4_to_fv129x256_aave.20150901.nc. What is "aave"? Weight generators can use about a dozen interpolation algorithms for regridding, and each has a shorthand name. For now, it is enough to know that the two most common algorithms are (first-order) conservative area-average regridding ("aave") and bilinear interpolation ("bilin" or "blin"). Hence this conservatively regrids dat_src.nc to dat_rgr.nc with first order accuracy:
...
Before looking into map-file generation in the next section, try a few ncremap features. For speed's sake, regrid only selected variables:
...
When an output directory is not specified, ncremap writes to the current working directory. When the output and input directories are the same, ncremap appends a string (based on the destination grid resolution) to each input filename to avoid name collisions. Finally, be aware that multiple-file invocations of ncremap execute serially in parallel by default. Power users will want to parallelize tune this as described in the section on "Intermediate Regridding".
...
The simplest regridding procedure applies an existing map-file to your data, as in the above example. If the desired map-file cannot be found, then you must create it. Creating a map-file requires a complete specification of both source and destination grids. The files that contain these grid specifications are called "grid-files". E3SM grid-files are available here (fxm: link). At most DOE High Performance Computing (HPC) centers these can also be found in my (@czender's) directory as ~zender/data/grids. Take a minute now to look there for the prototype problem grid-files, i.e., for FV 129x256 and ne30np4 grid-files.
...
Multiple grid-files also contain the string "ne30". These are either slightly different grids, or the same grids store in different formats meant for different post-processing tools. The different SE (and FV) grid types are described with figures and described here (https://acme-climate.atlassian.net/wiki/spaces/Docs/pages/34113147/Atmosphere+Grids). As explained there, most many people will want the "dual-grid" with pentagons. The correct grid-file for this is ne30np4_pentagons.091226.nc. Do not be tempted by SE grid-files named with "latlon".
...
ncremap -s ne30np4_pentagons.091226.nc -g 129x256_SCRIP.20150901.nc -m map_ne30np4_to_fv129x256_bilinnco.YYYYMMDD.nc
The map-file above is named with algtypalg_typ="bilinnco" because the ncremap default interpolation method is bilinearalgorithm is the first-order conservative NCO algorithm (NB: before NCO 4.9.1 the default algorithm was ESMF "bilin"). To re-create the "aave" map in the first example, invoke ncremap with the "-a conserve":
ncremap -a conserve -s ne30np4_pentagons.091226.nc -g 129x256_SCRIP.20150901.nc -m map_ne30np4_to_fv129x256_aave.YYYYMMDD.nc
This takes a few minutes, so save custom-generated map-files for future use. Computing weights to generate map-files is much more computationally expensive and time-consuming than regridding, i.e., than applying the weights in the map-file to the data. We will gloss over most options that weight-generators can take into consideration, because their default values often work well. One option worth knowing now is "-a". The invocation synonyms for "-a" are "--alg_typ", "--algorithm", and "--regrid_algorithm". These are listed in the square brackets in the self-help message that ncremap prints when it is invoked without argument, or with "--help":
...
zender@firn:~$
...
ncremap...-a
...
alg_typ
...
Algorithm
...
for
...
weight
...
generation
...
(default
...
bilinear)
...
[alg_typ,
...
algorithm,
...
regrid_algorithm]ESMF
...
algorithms:
...
bilinear|conserve|conserve2nd|nearestdtos|neareststod|patchTempest
...
algorithms:
...
tempest|se2fv_flx|se2fv_stt|se2fv_alt|fv2se_flx|fv2se_stt|fv2se_alt...
One valid option argument for each supported interpolation type is shown separated by vertical bars. The arguments shown have multiple synonyms that are equivalent. For example, "-a conserve" is equivalent to "-a aave" and to "--alg_typ=conservative". Use the longer option form for clarity and precision, and the shorter form for conciseness. The full list of synonyms, and the complete documentation, is at http://nco.sf.net/nco.html#alg_typ. The NCO algorithm "nco_conserve" is the default algorithm. "bilinear" and "conservative" are the mostcommonly-used algorithms that invoke ERWG. TR options are discussed below. Peruse the list of options now, though defer a thorough investigation until you reach the "Intermediate Regridding" section.
...