This page is devoted to instruction in ncremap. It describes steps necessary to create grids, and to regrid datasets between different grids with ncremap. Some of the simpler regridding options supported by ncclimo are also described at Generate, Regrid, and Split Climatologies (climo files) with ncclimo. This page describes those features in more detail, and other, more boutique features often useful for custom regridding solutions.
The Zen of Regridding
Most modern climate/weather-related research requires a regridding step in its workflow. The plethora of geometric and spectral grids on which model and observational data are stored ensures that regridding is usually necessary to scientific insight, especially the focused and variable resolution studies that E3SM models conduct. Why does such a common procedure seem so complex? Because a mind-boggling number of options are required to support advanced regridding features that many users never need. To defer that complexity, this HOWTO begins with solutions to the prototypical regridding problem, without mentioning any other options. It demonstrates how to solve that problem simply, including the minimal software installation required. Once the basic regridding vocabulary has been introduced, we solve the prototype problem when one or more inputs are "missing", or need to be created. The HOWTO ends with descriptions of different regridding modes and workflows that use features customized to particular models, observational datasets, and formats. The overall organization, including TBD sections (suggest others, or vote for prioritizing, below), is:
...
Inferral works well on important categories of grids for which ncremap can guess the missing grid information. In the absence of gridcell vertice information, ncremap examines the location of and spacing between gridcell centers and can often determine from these what type of grid a data-file (not a grid-file!) is stored on. A data-file simply means the filetype preferred for creation/distribution of modeled/observed data. Hence ncremap has the (original and unique, so far as we know) ability to infer all useful rectangular grid-types from data-files that employ the grid. The key constraint here is "rectangular", meaning the grid must be orthogonal (though not necessarily regularly spaced) in latitude and longitude. This includes all uniform angle grids, FV grids, and Gaussian grids. For curvilinear grids (including most swath data), ncremap infers the underlying grid to be the set of curves that bisect the curves created by joining the gridcell centers. This often works well for curvilinear grids that do not cross a pole. Inferral works for unstructured (i.e., 1D) grids only when the cell-bounds are stored in the datafile as described above. Hence inferral will not work on raw output from SE models.
...
The third option, the "sliding scale" approach, is slightly more complex and much more flexible than the all-or-nothing integral- or mean-preserving examples above. The sliding-scale refers to the fraction of the destination gridcell that must be overlapped by valid source gridcells. Users can specify the renormalization threshold weight rnr_thr which is the required valid fraction of destination gridcells with the renormalization threshold option "--rnr_thr=rnr_thr". The weight-application algorithm then ensures that valid values overlap at least the fraction rnr_thr of each destination gridcell for that gridcell to meet the threshold for a non-missing destination value. When rnr_thr is exceeded, the mean valid value in the valid area is placed in the destination gridcell. If the valid area covers less than rnr_thr, then the destination gridcell is assigned the missing value. Valid values of rnr_thr range from zero to one. A threshold weight of rnr_thr = 0.0 indicates that any amount (no matter how small) of valid data should be represented by its mean value on the output grid. Keep in mind though, that the actual valid area divides a sum (the area-weighted integral of the valid values), and values of zero or very near to zero in the divisor can lead to floating-point underflow and divide-by-zero errors. A threshold weight of rnr_thr = 1.0 indicates that the entire destination gridcell (to within machine precision) must be overlapped by valid data in order for that destination gridcell to be assigned a valid (non-missing) value. Threshold weights 0.0 < rnr_thr < 1.0 invoke the real-power of the sliding scale. Remote sensing classification schemes applied to L2 data may require, for example, valid retrievals over at least 50% of source pixels before assigning a valid value to a destination gridcell. This is equivalent to rnr_thr = 0.5. And so,
...
TempestRemap (TR) is the chief alternative to ERWG and NCO for regridding weight-generation. TR will replace is replacing ERWG as the default on-line weight generator in E3SMv2. Tempest algorithms, written by @paulullrich, have many numerical advantages described in papers and at Transition to TempestRemap for Atmosphere grids Verify . Verify that ncremap can access your Tempest installation as described in the above section on "Software Requirements" before trying the examples below. Once installed, TR can be as easy to use as ERWG or NCO with FV grid-files, e.g.,
| Code Block |
|---|
ncremap -a tempestfv2fv_flx -s ne30np4_pentagons.091226.nc -g 129x256_SCRIP.20150901.nc -m map_ne30np4_to_fv129x256_tempest.YYYYMMDD.nc |
This command, the same as shown in the "Create Map-file from Known Grid-files" section above, except using alg_typ='tempestfv2fv_flx', is a jumping-off point for understanding Tempest features and quirks. First, simply note that the ncremap interfaces for ERWG, NCO, and TR weight-generators are the same even though the underlying ERWG, NCO, and TR applications have different APIs. Second, Tempest accepts SCRIP-format grids (as shown) and Exodus-format grid-files, also stored in netCDF though typically with a '.g' suffix, e.g., ne30.g as described at Transition to TempestRemap for Atmosphere grids Exodus grid-files contain grid "connectivity" and other information required to optimally treat advanced grids like SE. Hence this also works
| Code Block |
|---|
ncremap -a tempestse2fv_flx -s ne30.g -g 129x256_SCRIP.20150901.nc -m map_ne30np4_to_fv129x256_tempest.YYYYMMDD.nc |
This produces subtly different weights because ne30.g encodes the SE ne30np4 grid-specification, not its dual-grid FV representation. TR generates superior weights when instructed to use an algorithm optimized for the type of remapped variable and the grid representation as described below. In the The above example , employs the best recommended algorithm choice to remap fluxes on the SE grid to the FV destination grid would be
...
, i.e., se2fv_flx
...
.
...
The exact switches options that NCO invokes for a specific TR algorithm like se2fv_flx can be discovered in multiple ways: First, all TR options that NCO employs are recommended on the Transition to TempestRemap for Atmosphere grids; Second, the NCO Users Guide documents TR options at http://nco.sf.net/nco.html#tr; Third, the options --dbg_lvl=1 or --dbg_lvl=2 cause ncremap to print the sub-commands it executes, including the TR commands with options. Experiment, intercompare, and find the algorithm that works best for your purposes. Advanced users may be interested in the quantitative evaluations of the quality of the weights in the map-file provided by the map-checker (ncks --chk_map map.nc) described below.
...
File-level parallelism accelerates throughput when regridding multiple files in one ncremap invocation, and has no effect when only one file is to be regridded. Note that the ncclimo and ncremap semantics for selecting file-level parallelism are identical, though their defaults differ (Background mode for ncclimo and Serial mode for ncremap). Select the desired mode with the argument to --par_typ=par_typ. Explicitly select Background mode with par_typ values of bck, background, or Background. The values mpi or MPI select MPI mode, and the srl, serial, Serial, nil, and none will all select Serial mode (which disables file-level parallelism, though still allows intra-file OpenMP parallelism).
...