This page is devoted to instruction in ncremap. It describes steps necessary to create grids, and to regrid datasets between different grids with ncremap. Some of the simpler regridding options supported by ncclimo are also described at Generate, Regrid, and Split Climatologies (climo files) with ncclimo. This page describes those features in more detail, and other, more boutique features often useful for custom regridding solutions.
The Zen of Regridding
Most modern climate/weather-related research requires a regridding step in its workflow. The plethora of geometric and spectral grids on which model and observational data are stored ensures that regridding is usually necessary to scientific insight, especially the focused and variable resolution studies that E3SM models conduct. Why does such a common procedure seem so complex? Because a mind-boggling number of options are required to support advanced regridding features that many users never need. To defer that complexity, this HOWTO begins with solutions to the prototypical regridding problem, without mentioning any other options. It demonstrates how to solve that problem simply, including the minimal software installation required. Once the basic regridding vocabulary has been introduced, we solve the prototype problem when one or more inputs are "missing", or need to be created. The HOWTO ends with descriptions of different regridding modes and workflows that use features customized to particular models, observational datasets, and formats. The overall organization, including TBD sections (suggest others, or vote for prioritizing, below), is:
...
The terminology of this section can be confusing. The most important point to clarify is that the treatments of missing data described here are independent of the regridding algorithm (and weight-generator) which is specified by alg_typ which defines . alg_typ only defines the algorithm used to generate the weights contained in a map-file, not how to apply those weights. The generation algorithm may or may not be conservative. ncremap applies the map-file weights to regrid (i.e., regrids) to data-files that may contain fields with spatio-temporally missing values, such as AODVIS in EAM, or many other , all fields with ocean gridpoints in ELM output, most fields in MPAS-Seaice, and many fields in satellite-retrieved datasets. Unless the locations of a field's (like AODVIS) missing values were supplied as a mask to the weight-generator algorithm at map-creation time, the generated map-file weights cannot and will not automatically take into account that some of the source gridcells contain invalid (aka missing) values. This section describes two three options for how the weight-applicator (not weight-generator) is to reconstruct values in destination gridcells affected by missing source gridcells. One option preserves the integral value of the input gridcells and is called "integral-preserving" or "conservative" because it is first-order conservative in the affected gridcells , and locally, as well as globally conservative. At the other extreme is the option to preserve ("renormalized") is not. These approaches are independent of the weight-generation algorithm defined locally) the gridcell mean of the input values in the affected output gridcells. This option is called "mean-preserving" or "renormalized" and is not conservative. The third option allows the user to specify a sliding scale between the integral- and mean-preserving options, which is to say it is a more general form of renormalization. Once again, these weight-application approaches are independent of the weight-generation algorithm defined by alg_typ.
Conservative regridding weight-generation is, for first-order accurate algorithms, a straightforward procedure of identifying gridcell overlap and apportioning values correctly from source to destination. The absence of valid values (or presence of missing values) forces a choice on how to construct destination gridcell values where some but not all contributing source cells are valid. ncremap supports two three distinct approaches: "conservative" and "renormalized"Integral-preserving", "Mean-preserving", and "Sliding Scale". The conservative integral-preserving approach uses all valid data from the input grid on the output grid once and only once. Destination cells receive the weighted valid values of the source cells. This is a first-order conservative procedure, and will, in conjunction with a weights from conservative regridding algorithm, conserve and preserve the local and global integrals of the source and destination fields. The renormalized mean-preserving approach divides the destination value by the sum of the valid weights. This preserves and extends the mean of the valid input values , though extended to throughout the entire destination gridcell. Thus missing-data renormalization is equivalent to extrapolating In other words, it extrapolates valid data to missing regions while preserving the mean valid data. Input and output integrals are unequal and renormalized mean-preserving regridding is not conservative and does not preserve the integrals when missing data are involved. Furthermore, mean-preserving regridding only preserves the local (i.e., gridcell-level) mean of the input data, not the global mean. Both approaches, conservative and re-normalized (or integral-preserving and mean-preserving (aka conservative and re-normalized) produce identical answers when no missing data maps to the destination gridcell. Before explaining the nuance of the third approach ("sliding-scale"), we demonstrate the symmetric ways of invoking integral- or mean-preserving regridding:
ncremap -m map.nc dat_src.nc dat_rgr.nc # Integral-preserving treatment of missing data is used by default
ncremap --preserve=integral -m map.
...
nc dat_src.nc dat_rgr.nc # Explicitly specified integral-preserving treatment
ncremap --preserve=mean -m map.nc dat_src.nc dat_rgr.nc # Explicitly specified mean-preserving treatment
By default, ncremap implements the integral-preserving, conservative approach because it has useful properties, is simpler to understand, and requires no additional parameters. However, this approach will often produce unrealistic gridpoint values (e.g., a temperature of 10 K) , and a conservative treatment of missing data often produces such values along coastlines or near data gaps where state variables are regridded to/from small fractions of a gridcell. Renormalization . The mean-preserving approach solves this problem, yet incurs others, like non-conservation. The mean-preserving approach (aka re-normalization) ensures that the output values are physically consistent and do not stick-out like sore thumbs in a plot, although the integral of their value times area is not conserved.
By default, ncremap implements the conservative approach because it has useful properties, is simpler to understand, and requires no additional parameters. To employ the renormalization algorithm instead, supply the threshold weight for valid destination values with "--rnr_thr=rnr_thr". Valid values must cover at least the fraction rnr_thr of the destination gridcell to meet the threshold for a non-missing destination value. When rnr_thr is exceeded, the mean valid value is renormalized by the valid area and placed in the destination gridcell. If the valid area covers less than rnr_thr, then the destination gridcell is assigned the missing value. Valid values of rnr_thr range from zero to one. Keep in mind though, that this threshold is potentially a divisor, and values of zero or very near to zero can lead to floating-point underflow and divide-by-zero errors. For convenience, ncremap permits users to specify a threshold weight of rnr_thr = 0.0. This indicates that any valid data should be represented and renormalized on the output grid.
...