This page is devoted to instruction in ncremap. It describes steps necessary to create grids, and to regrid datasets between different grids with ncremap. Some of the simpler regridding options supported by ncclimo are also described at Generate, Regrid, and Split Climatologies (climo files) with ncclimo. This page describes those features in more detail, and other, more boutique features often useful for custom regridding solutions.
The Zen of Regridding
Most modern climate/weather-related research requires a regridding step in its workflow. The plethora of geometric and spectral grids on which model and observational data are stored ensures that regridding is usually necessary to scientific insight, especially the focused and variable resolution studies that E3SM models conduct. Why does such a common procedure seem so complex? Because a mind-boggling number of options are required to support advanced regridding features that many users never need. To defer that complexity, this HOWTO begins with solutions to the prototypical regridding problem, without mentioning any other options. It demonstrates how to solve that problem simply, including the minimal software installation required. Once the basic regridding vocabulary has been introduced, we solve the prototype problem when one or more inputs are "missing", or need to be created. The HOWTO ends with descriptions of different regridding modes and workflows that use features customized to particular models, observational datasets, and formats. The overall organization, including TBD sections (suggest others, or vote for prioritizing, below), is:
...
The terminology of this section can be confusing. The most important point to clarify is that the treatments of missing data described here are independent of the regridding algorithm (and weight-generator) specified by alg_typ. alg_typ which defines the algorithm used to generate the weights contained in a map-file. That , not how to apply those weights. The generation algorithm may or may not be conservative. ncremap applies the map-file weights to regrid data-files that may contain fields with missing values, such as AODVIS in EAM, or many other fields in satellite-retrieved datasets. Unless the locations of a field's (like AODVIS) missing values were supplied as a mask to the weight-generator algorithm, the generated map-file weights cannot automatically take into account that some of the source gridcells contain invalid (aka missing) values. This section describes two options for how the weight-applicator (not weight-generator) is to reconstruct values in destination gridcells affected by missing source gridcells. One option is called "conservative" because it is first-order conservative in the affected gridcells, and the other option ("renormalized") is not. These approaches are independent of the weight-generation algorithm defined by alg_typ.
Conservative regridding is, for first-order accurate algorithms, a straightforward procedure of identifying gridcell overlap and apportioning values correctly from source to destination. The absence of valid values (or presence of missing values) forces a choice on how to construct destination gridcell values where some but not all contributing source cells are valid. ncremap supports two distinct approaches: "conservative" and "renormalized". The conservative approach uses all valid data from the input grid on the output grid once and only once. Destination cells receive the weighted valid values of the source cells. This is a first-order conservative procedure, and will, in conjunction with a conservative regridding algorithm, conserve and preserve the global integrals of the source and destination fields. The renormalized approach divides the destination value by the sum of the valid weights. This produces values equal to preserves the mean of the valid input values, though extended to the entire destination gridcell. Thus missing-data renormalization is equivalent to extrapolating valid data to missing regions while preserving the mean. Input and output integrals are unequal and renormalized regridding is not conservative and does not preserve the integrals when missing data are involved.. Both approaches, conservative and re-normalized (or integral-preserving and mean-preserving) produce identical answers when no missing data maps to the destination gridcell.
Renormalization is useful because it solves some problems, like physically unrealistic values, yet incurs others, like non-conservation. Many researchers eschew unrealistic gridpoint values (e.g., a temperature of 10 K), and a conservative treatment of missing data often produces such values along coastlines or near data gaps where state variables are regridded to/from small fractions of a gridcell. Renormalization ensures that the output values are physically consistent, although the integral of their value times area is not conserved.
By default, ncremap implements the conservative approach because it has useful properties, is simpler to understand, and requires no additional parameters. To employ the renormalization algorithm instead, supply the threshold weight for valid destination values with "--rnr_thr=rnr_thr". Valid values must cover at least the fraction rnr_thr of the destination gridcell to meet the threshold for a non-missing destination value. When rnr_thr is exceeded, the mean valid value is renormalized by the valid area and placed in the destination gridcell. If the valid area covers less than rnr_thr, then the destination gridcell is assigned the missing value. Valid values of rnr_thr range from zero to one. Keep in mind though, that this threshold is potentially a divisor, and values of zero or very near to zero can lead to floating-point underflow and divide-by-zero errors. For convenience, ncremap permits users to specify a threshold weight of rnr_thr = 0.0. This indicates that any valid data should be represented and renormalized on the output grid.
...
In practice, it may make sense to use the default conservative treatment with conservative regridding, and the renormalized treatment with other (non-conservative) regridding algorithms such as bilinear interpolation or nearest-neighbor. Another consideration is whether the fields being regridded are fluxes or state variables. For example, temperature (unlike heat) and concentrations (amount per unit volume) are not physically conserved quantities under areal-regridding so it often makes sense to interpolate them in a non-conservative fashion, to preserve their fine-scale structure. Few researchers can digest the unphysical values of temperature that the conservative treatment produces in regions rife with missing values. On the other hand are , mass and energy fluxes , which should be physically conserved under areal-regridding. One should consider both the type of field and its conservation properties when choosing a missing value treatment.
...