Page Comparison

This page is devoted to instruction in ncremap. It describes steps necessary to create grids, and to regrid datasets between different grids with ncremap. Some of the simpler regridding options supported by ncclimo are also described at Generate, Regrid, and Split Climatologies (climo files) with ncclimo. This page describes those features in more detail, and other, more boutique features often useful for custom regridding solutions.

The Zen of Regridding

Most modern climate/weather-related research requires a regridding step in its workflow. The plethora of geometric and spectral grids on which model and observational data are stored ensures that regridding is usually necessary to scientific insight, especially the focused and variable resolution studies that E3SM models conduct. Why does such a common procedure seem so complex? Because a mind-boggling number of options are required to support advanced regridding features that many users never need. To defer that complexity, this HOWTO begins with solutions to the prototypical regridding problem, without mentioning any other options. It demonstrates how to solve that problem simply, including the minimal software installation required. Once the basic regridding vocabulary has been introduced, we solve the prototype problem when one or more inputs are "missing", or need to be created. The HOWTO ends with descriptions of different regridding modes and workflows that use features customized to particular models, observational datasets, and formats. The overall organization, including TBD sections (suggest others, or vote for prioritizing, below), is:

...

ncremap -a conserve -s ne30np4_pentagons.091226.nc -g 129x256_SCRIP.20150901.nc -m map_ne30np4_to_fv129x256_aave.YYYYMMDD.nc # EAMv1
ncremap -a conserve -s ne30pg2.nc -g cmip6_180x360_scrip.20181001.nc -m map_ne30pg2_to_cmip6_180x360_ncoaave.YYYYMMDD.nc # EAMv2

This takes a few minutes, so save custom-generated map-files for future use. Computing weights to generate map-files is much more computationally expensive and time-consuming than regridding, i.e., than applying the weights in the map-file to the data. We will gloss over most options that weight-generators can take into consideration, because their default values often work well. One option worth knowing now is -a. The invocation synonyms for -a are --alg_typ, --algorithm, and --regrid_algorithm. These are listed in the square brackets in the self-help message that ncremap prints when it is invoked without argument, or with --help:

...

Before we describe what the inferral procedure does, here is an example that demonstrates how easy it is. You can regrid an SE (or FV) dataset from our prototype example to the same grid as an evaluation dataset. Pick any 2D (ie.eg., latxlon MxN latitude-by-longitude) dataset to compare the SE data E3SM simulations to. Inferral uses the grid information in the evaluation dataset, which is already on the desired destination grid, to create the (internally generated) destination grid-file. Supply ncremap with any dataset on the desired destination grid (dat_dst.nc) with "-d" (for "dataset") instead of "-g" (for "grid"):

ncremap -a bilin -s ne30np4_pentagons.091226.nc -d dat_dst.nc -m map_ne30np4_to_MxN_bilin.YYYYMMDD.nc # EAMv1
ncremap -a bilin -s ne30pg2.nc -d dat_dst.nc -m map_ne30pg2_to_1x1MxN_bilin.YYYYMMDD.nc # EAMv2

This tells ncremap to infer the destination grid-file and to use it to generate the desired map-file, named with the supposed destination resolution (here, 1x1 degree MxN degrees or gridcells). To archive the inferred destination grid for later use, supply a name for the grid with "-g":

ncremap -s ne30np4_pentagons.091226.nc -d dat_dst.nc -g grd_dst.nc -m map_ne30np4_to_1x1_bilinnco.YYYYMMDD.nc # Requires NCO version >= 4.7.6
ncremap -s ne30pg2.nc -d dat_dst.nc -g grd_dst.nc -m map_ne30pg2_to_MxN_nco.YYYYMMDD.nc # EAMv2

One can infer a grid without having to regrid anything. Supply ncremap with a data-template file (dat.nc) and a grid-file name (grd.nc). Since there are no input files to regrid, ncremap exits after inferring the grid-file:

...

Grid-inferral is easier to try than manual grid-generation, and will work if the data file contains the necessary information. The only data needed to construct a SCRIP grid-file are the vertices of each gridcell. The gridcell vertices define the gridcell edges and these in turn define the gridcell area which is equivalent to the all-important weight parameter necessary to regrid data. Of course the gridcell vertices must be stored with recognizable names and/or metadata indicators. The Climate & Forecast (CF) metadata convention calls the gridcell vertices the "cell-bounds". Coordinates (like latitude and longitude) usually store cell-center values, and should, according to CF, have "bounds" attributes whose values point to variables (e.g., "lat_bounds" or "lon_vertices") that contain the actual vertices. Relatively few datasets "in the wild" contain gridcell vertices, though the practice is, happily, on the rise. Formally SE models have nodal points with weights without any fixed perimeter or vertices assigned to the nodes, so the lack of vertices in SE model output is a feature, not an oversight. The dual-grid (referenced above) addresses this by defining "pretend" gridcell vertices for each nodal point so that an SE dataset can be treated like an FV dataset. However, dual-grids are difficult to generate, and may not exist or be accurate for many SE grids. In that case, ncremap cannot infer the grid (because the vertices are unknown) and one needs to use a different package (such as TempestRemap, below) to construct the grid-file and the mapping weights.

Inferral works well on important categories of grids for which ncremap can guess the missing grid information. In the absence of gridcell vertice information, ncremap examines the location of and spacing between gridcell centers and can often determine from these what type of grid a data-file (not a grid-file!) is stored on. A data-file simply means the filetype preferred for creation/distribution of modeled/observed data. Hence ncremap has the (original and unique, so far as we know) ability to infer all useful rectangular grid-types from data-files that employ the grid. The key constraint here is "rectangular", meaning the grid must be orthogonal (though not necessarily regularly spaced) in latitude and longitude. This includes all uniform angle grids, FV grids, and Gaussian grids. For curvilinear grids (including most swath data), ncremap infers the underlying grid to be the set of curves that bisect the curves created by joining the gridcell centers. This often works well for curvilinear grids that do not cross a pole. Inferral works for unstructured (i.e., 1D) grids only when the cell-bounds are stored in the datafile as described above. Hence inferral will not work on raw output from SE models.

...

Here the user provides only data files (no grid- or map-files) yet still obtains regridded output! The first positional (i.e., not immediately preceded by an option indicator) argument (dat_src.nc) is interpreted as the data to regrid, and the second positional argument (dat_rgr.nc) is the output name. The -d argument is the name of any dataset (dat_dst.nc) on the desired destination grid. ncremap infers the source grid from dat_src.nc, then infers the destination grid from dat_dst.nc, then generates weights (with the default algorithm since none is specified) and creates a map-file that it uses to regrid dat_src.nc to dat_rgr.nc. No grid-file or map-file names were specified (with -g or -m) so both grid-files and the map-file are generated internally in temporary locations and erased after use.

...

ncremap -M -d dat_dst.nc -I drc_in -O drc_rgr
ncremap --mlt_map --dst_fl=dat_dst.nc --drc_in=drc_in --drc_rgr=drc_rgr # Long-options for clarity

Prototypical Regridding IV: Manual Grid-file Generation

...