This page is devoted to instruction in ncremap. It describes steps necessary to create grids, and to regrid datasets between different grids with ncremap. Some of the simpler regridding options supported by ncclimo are also described at Generate, Regrid, and Split Climatologies (climo files) with ncclimo. This page describes those features in more detail, and other, more boutique features often useful for custom regridding solutions.
The Zen of Regridding
Most modern climate/weather-related research requires a regridding step in its workflow. The plethora of geometric and spectral grids on which model and observational data are stored ensures that regridding is usually necessary to scientific insight, especially the focused and variable resolution studies that E3SM models conduct. Why does such a common procedure seem so complex? Because a mind-boggling number of options are required to support advanced regridding features that many users never need. To defer that complexity, this HOWTO begins with solutions to the prototypical regridding problem, without mentioning any other options. It demonstrates how to solve that problem simply, including the minimal software installation required. Once the basic regridding vocabulary has been introduced, we solve the prototype problem when one or more inputs are "missing", or need to be created. The HOWTO ends with descriptions of different regridding modes and workflows that use features customized to particular models, observational datasets, and formats. The overall organization, including TBD sections (suggest others, or vote for prioritizing, below), is:
...
This takes a few minutes, so save custom-generated map-files for future use. Computing weights to generate map-files is much more computationally expensive and time-consuming than regridding, i.e., than applying the weights in the map-file to the data. We will gloss over most options that weight-generators can take into consideration, because their default values often work well. One option worth knowing now is "-a". The invocation synonyms for "-a" are "--alg_typ", "--algorithm", and "--regrid_algorithm". These are listed in the square brackets in the self-help message that ncremap prints when it is invoked without argument, or with "--help":
zender@firn:~$ ncremap
...
-a alg_typ Algorithm for weight generation (default nco_con) [alg_typ, algorithm, regrid_algorithm]
ESMF algorithms: bilinear|conserve|conserve2nd|nearestdtos|neareststod|patch
NCO algorithms: nco_con (1st order conservative algorithm similar to "conserve")
Tempest algorithms: fv2fv|fv2fv_flx|fv2fv_stt|fv2se_flx|fv2se_stt|fv2se_alt|se2fv_flx|se2fv_stt|se2fv_alt|se2se|tempest
...
One valid option argument for each supported interpolation type is shown separated by vertical bars. The arguments shown have multiple synonyms that are equivalent. For example, "-a conserve" is equivalent to "-a aave" and to "--alg_typ=conservative". Use the longer option form for clarity and precision, and the shorter form for conciseness. The full list of synonyms, and the complete documentation, is at http://nco.sf.net/nco.html#alg_typ. The NCO algorithm "nco_conserve" is the default algorithm. Commonly-used algorithms that invoke ERWG are "bilinear" and "conservative". TR options are discussed below. Peruse the list of options now, though defer a thorough investigation until you reach the "Intermediate Regridding" section.
Prototypical Regridding III: Infer Grid-file from Data-file
Thus far we have explained how to apply a map-file to data, and how, if necessary, to generate a map-file from known grids. What if there is no map-file and the source or the destination grid-files (or both) are unavailable? Often, one knows the basic information about a grid (e.g., resolution) but lacks the grid-file that contains the complete information for that grid geometry. In such cases, one must create the grid-file via one of two methods. First, one can let ncremap attempt to infer the grid-file from a data file known to be on the desired grid. This procedure is called "inferral" and is fairly painless. Second, one can feed ncremap all the required parameters (for rectangular grids only) and it will generate a grid-file. This requires a precise specification of the grid geometry, and will be covered the sub-section on "Manual Grid-file Generation".
...
Intermediate Regridding II: TempestRemap
TempestRemap (TR) is the chief alternative to ERWG and NCO for regridding weight-generation, and is sleighted for increased adoption in E3SM. TR will replace ERWG as the default on-line weight generator in E3SMv2. Tempest algorithms, written by @paulullrich, have many numerical advantages described in papers and at https://acme-climate.atlassian.net/wiki/spaces/Docs/pages/178848194/Transition+to+TempestRemap+for+Atmosphere+grids. Tempest lacks pre-built binary package support under most OSs and on most DOE machines so users must often download and install it by following the instructions at https://github.com/ClimateGlobalChange/tempestremap. Verify . Verify that ncremap can access your Tempest installation as described in the above section on "Software Requirements" before trying the examples below. Once installed, Tempest TR can be as easy to use as ERWG or NCO with SCRIP FV grid-files, e.g.,
ncremap -a tempest -s ne30np4_pentagons.091226.nc -g 129x256_SCRIP.20150901.nc -m map_ne30np4_to_fv129x256_tempest.YYYYMMDD.nc
This command, the same as shown in the "Create Map-file from Known Grid-files" section above, except using algtyp alg_typ='tempest', is a jumping-off point for understanding Tempest features and quirks. First, simply note that the ncremap interface interfaces for ERWG and Tempest , NCO, and TR weight-generators are the same even though the underlying ERWG, NCO, and Tempest TR applications have different APIs. Second, Tempest accepts SCRIP-format grids (as shown) and Exodus-format grid-files, also stored in netCDF though typically with a '.g' suffix, e.g., ne30.g as described on the Transition-to-Tempest page just mentioned. Exodus grid-files contain grid "connectivity" and other information required to optimally treat advanced grids like SE. Hence this also works
...
This produces subtly different weights because ne30.g encodes the SE ne30np4 grid-specification, not its dual-grid FV representation. TR generates superior weights when instructed to use an algorithm optimized for the type of remapped variable and the grid representation as described below. In the above example, the best algorithm choice to remap fluxes on the SE grid to the FV destination grid would be
ncremap -a se2fv_flx -s ne30.g -g 129x256_SCRIP.20150901.nc -m map_ne30np4_to_fv129x256_tempest.YYYYMMDD.nc
Experiment, intercompare, and find what the algorithm that works best for your purposes. As mentioned in the Transition-to-Tempest page, the Tempest overlap-mesh generator expects to be sent the two grid-files in the order smaller then larger (ERWG has and NCO have no corresponding restriction). For example, Tempest considers the global ocean to be a smaller domain than the global atmosphere since it covers less area (due to masked points). Hence ncremap must be told when the source grid-file covers a larger domain than the destination. Do this with the "--a2o" or "--l2s" switch (for "atmosphere-to-ocean" or "large-to-small", respectively):
...
Finally, the Tempest-transition page describes six eight specific global FV<->SE mappings that optimize different combinations of accuracy, conservation, and monotonicity desirable for remapping flux-variables (flx), state-variables (stt), and an alternative (alt) mapping. A plethora of boutique options and switches control the Tempest weight-generation algorithms for these six cases. To simplify their invocation, ncremap names these six eight algorithms fv2se_stt, fv2se_flx, fv2se_alt, fv2fv_flx, fv2fv_stt, se2fv_stt, se2fv_flx, and se2fv_alt, and se2se. E3SM maps with these algorithms have adopted the suffixes "mono" (for both se2fv_flx and fv2se_alt), "highorder" (se2fv_stt, fv2se_stt), "intbilin" (se2fv_alt), and "monotr" (fv2se_flx). The six relevant Tempest map-files can be generated with
ncremap -a fv2se_flx -s ocean.RRS.30-10km_scrip_150722.nc -g ne30.g -m map_oRRS30to10_to_ne30np4_monotr.20180901.nc
ncremap -a fv2se_stt -s ocean.RRS.30-10km_scrip_150722.nc -g ne30.g -m map_oRRS30to10_to_ne30np4_highorder.20180901.nc
ncremap -a fv2se_alt -s ocean.RRS.30-10km_scrip_150722.nc -g ne30.g -m map_oRRS30to10_to_ne30np4_mono.20180901.nc
ncremap --l2s -a se2fv_flx -s ne30.g -g ocean.RRS.30-10km_scrip_150722.nc -m map_ne30np4_to_oRRS30to10_mono.20180901.nc
ncremap --l2s -a se2fv_stt -s ne30.g -g ocean.RRS.30-10km_scrip_150722.nc -m map_ne30np4_to_oRRS30to10_highorder.20180901.nc
ncremap --l2s -a se2fv_alt -s ne30.g -g ocean.RRS.30-10km_scrip_150722.nc -m map_ne30np4_to_oRRS30to10_intbilin.20180901.nc
ncremap -a se2se -s ne30.g -g ne120.g -m map_ne30np4_to_ne120np4_se2se.20180901.nc
ncremap -a fv2fv_flx -s ocean.RRS.30-10km_scrip_150722.nc -g cmip6_180x360_scrip.20181001.nc -m map_oRRS30to10_to_180x360_mono.20180901.nc
ncremap -a fv2fv_stt -s ocean.RRS.30-10km_scrip_150722.nc -g cmip6_180x360_scrip.20181001.nc -m map_oRRS30to10_to_180x360_highorder.20180901.nc
These maps, applied to appropriate flux and state variables, should exactly reproduce the online remapping in the E3SM v2/v3 coupler. However, explicitly generating all six standard maps this way is not recommended because ncremap includes an MWF-mode (for "Make All Weight Files") described below. MWF-mode generates and names, with one command and in a self-consistent manner, all combinations of E3SM global atmosphere<->ocean maps with both for ERWG and Tempest.
Intermediate Regridding III: Parallelism (TBD)
...