This page is devoted to instruction in ncremap. It describes steps necessary to create grids, and to regrid datasets between different grids with ncremap. Some of the simpler regridding options supported by ncclimo are also described at Generate, Regrid, and Split Climatologies (climo files) with ncclimo. This page describes those features in more detail, and other, more boutique features often useful for custom regridding solutions.
The Zen of Regridding
Most modern climate/weather-related research requires a regridding step in its workflow. The plethora of geometric and spectral grids on which model and observational data are stored ensures that regridding is usually necessary to scientific insight, especially the focused and variable resolution studies that E3SM models conduct. Why does such a common procedure seem so complex? Because a mind-boggling number of options are required to support advanced regridding features that many users never need. To defer that complexity, this HOWTO begins with solutions to the prototypical regridding problem, without mentioning any other options. It demonstrates how to solve that problem simply, including the minimal software installation required. Once the basic regridding vocabulary has been introduced, we solve the prototype problem when one or more inputs are "missing", or need to be created. The HOWTO ends with descriptions of different regridding modes and workflows that use features customized to particular models, observational datasets, and formats. The overall organization, including TBD sections (suggest others, or vote for prioritizing, below), is:
...
These maps, applied to appropriate flux and state variables, should exactly reproduce the online remapping in the E3SM v2/v3 coupler. However, explicitly generating all standard maps this way is not recommended because ncremap includes an MWF-mode (for "Make All Weight Files") described below. MWF-mode generates and names, with one command and in a self-consistent manner, all combinations of E3SM global atmosphere<->ocean maps for ERWG and Tempest.
Intermediate Regridding III: MOAB/mbtempest support
Weight-generator computations (e.g., finding grid intersections) increase non-linearly with the grid size, so the largest grids are most efficiently computed with parallel algorithms. ERWG has long supported distributed computation in an MPI-environment, and NCO has always supported multi-threaded weight computation via OpenMP. A growing subset of TempestRemap algorithms have now been ported to the parallel MOAB (Mesh Oriented datABase) tool mbtempest (aka MBTR). ncremap can invoke the MOAB regridding toolchain as of NCO version 5.0.2 from September, 2021. The "spoiler" answer to "How do I get ncremap to invoke mbtempest?" is simple: Add the --mpi_nbr=mpi_nbr option to an ncremap command that already calls a TempestRemap algorithm. However, MOAB involves complex, MPI-enabled software, and support for it is continually changing and subject to important limitations. Read on to understand what to currently expect.
First, MOAB requires an MPI environment to perform well. Invoking MOAB (i.e., using --mpi_nbr=mpi_nbr with a TR algorithm name) in a non-MPI environment will result in an error. One can easily obtain MPI-enabled MOAB environment with Conda. For example, install Conda MPI versions of the MOAB package with
conda install -c conda-forge moab=5.3.0=mpich_tempest esmf
Please ensure you have the latest version of ERWG, MOAB, and/or TempestRemap before reporting any related problems to NCO.
This section is intentionally placed after the TR section because mbtempest re-uses the TR algorithm names described in the previous section. For example, ncremap invokes TR to generate weights to remap fluxes from FV to FV grids when invoked with
ncremap -a fv2fv_flx --src_grd=src.g --dst_grd=dst.nc -m map.nc # Invoke TR
Add the --mpi_nbr option and ncremap will instead invoke the MOAB toolchain to compute weights for any TempestRemap algorithm (otherwise the TR toolchain would be used):
ncremap --mpi_nbr=8 -a fv2fv_flx --src_grd=src.g --dst_grd=dst.nc -m map.nc # Invoke MOAB
Although mapping weights generated by MOAB and TempestRemap use the same numerical algorithms, they are likely to produce slightly different weights due to round-off differences. MOAB is heavily parallelized and computes and adds terms together in an unpredictable order compared to the serial TempestRemap.
Transparently to the user, ncremap supplies the selected weight generator with the recommended options, which can be quite complex. For the preceding example, ncremap either invokes TempestRemap's GenerateOverlapWeights with the boutique options --in_type fv --in_np 1 --out_type fv --out_np 1 --correct_areas that E3SM recommends for conservative and monotone remapping of fluxes, or it invokes multiple components of the MOAB toolchain:
mbconvert -B -o PARALLEL=WRITE_PART -O PARALLEL=BCAST_DELETE -O PARTITION=TRIVIAL -O PARALLEL_RESOLVE_SHARED_ENTS "src.g" "src.h5m"mbconvert -B -o PARALLEL=WRITE_PART -O PARALLEL=BCAST_DELETE -O PARTITION=TRIVIAL -O PARALLEL_RESOLVE_SHARED_ENTS "dst.nc" "dst.h5m"mbpart 8 --zoltan RCB "src.h5m" "src_8p.h5m"mbpart 8 --zoltan RCB --recompute_rcb_box --scale_sphere --project_on_sphere 2 "dst.h5m" "dst_8p.h5m"mpirun -n 8 mbtempest --type 5 --weights --load "src_8p.h5m" --load "dst_8p.h5m" --method fv --order 1 --method fv --order 1 --file "map.nc"
Once again, the MOAB toolchain should produce a map-file identical (to rounding precision) to one produced by TR. When speed matters (i.e., large grids), and the algorithm is supported (i.e., fv2fv_flx), invoke MOAB, otherwise invoke TR.
This section comes before the parallelism section because mbtempest supports MPI-enabled parallelism that is distinct from the ncremap workflow parallelism described in the next section.
Caveat lector: As of September, 2021 Weights generated by MOAB (version 5.3.0 and earlier) are only trustworthy for the fv2fv_flx algorithm. The options for all other algorithms are implemented as indicated though they should be invoked for testing purposes only. High order and spectral element map algorithms are not fully implemented and will produce unreliable results. MOAB anticipates supporting more TempestRemap algorithms in the future. Always use the map-checker to test maps before use, e.g., with ncks --chk_map map.nc.
Intermediate Regridding IV: Parallelism
ncremap can exploit three types of parallelism: multiples nodes in a cluster, multiple simultaneous file regriddings on a single node, and multiple simultaneous variables regridded within a single file. Each level of parallelism reduces the wallclock time to complete the regridding workflow at the expense of increase resource requirements.
...
First, use the right tool. ncremap can regrid an initial conditions (IC) file, both vertically and horizontally. However, generating scientifically validated IC files for new model resolutions is best done with a lengthy workflow (https://acme-climate.atlassian.net/wiki/spaces/ED/pages/872579110/Adding+support+for+new+grids) in which regridding plays a relatively small role. That said, ncremap is a good tool to place an atmospheric state onto a new grid where it can then be nudged into a valid IC file, or to place the contents of an IC file on a rectangular grid (as shown below) where it is easier to plot. Regridding atmosphere IC files was straightforward until E3SM v2 when the atmosphere separated the dynamical and physical grids. This resulted in IC files containing two sets of grid variables, so now we must regrid v2 IC files with two successive invocations of ncremap:
ncremap --map=map_ne30pg2_to_cmip6_180x360_nco.20200901.nc foo.eam.i.2001-01-01-00000.nc foo.ncncremap --map=map_ne30np4_to_cmip6_180x360_nco.20200901.nc foo.nc foo.eam.i.2001-01-01-00000.rgr.nc
The first command remaps the variables on the PG2 physics grid to the desired output grid. This is straightforward since the physics grid variables (SICTHK, ICEFRAC, TS1...) use ncol (the ncremap default) as the horizontal coordinate. The second command remaps the remaining variables, stored on the NP4 dynamics grid with horizontal dimension ncol_d, to the same output grid. The second invocation automatcally regrids all the variables on the dynamics grid (with the ncol_d dimension) because the intermediate file foo.nc no longer has an ncol dimension.
...
ncremap -R '--rgr col_nm=ncol_d' --map=map_ne30np4_to_cmip6_180x360_nco.20200901.nc foo.eam.i.2001-01-01-00000.nc foo.rgr.nc
Advanced Regridding VII: Fixing Grid Files to Work as Intended
...