This page documents the recommended mapping files specifications for running E3SM and analyzing the output, which should be used for all E3SMv2 configurations. This will ease the transition to inline mapping in E3SMv2 with the MOAB based coupler which has inline support for Tempest algorithms.  

For each map (i.e. FV-FV mono, FV-SE monotr), see the sections below on naming conventions and specific command line options used to produce each map.  

Often map file algorithm choices dont have much of an impact on the solution, but occasionally they do.   Some example problems due to mapping file issues:   

• Here is a great example showing the noise that can be generated when an aave map is used for coarse-to-fine mapping of a non-flux variables:   /wiki/spaces/ECG/pages/2350776353 , also see the many plots here: https://github.com/E3SM-Project/E3SM/issues/5242
• Another example showing blockyness from conservative maps when upsampling:  /wiki/spaces/NGDNA/pages/3530194969
• When downsampling, interpolation algorithms (like bilinear) have larger aliasing errors as compared to area averaged maps: Downsampling algorithm comparison
• "Speckling":  inconsistencies between the land/ocean fraction and maps to the atmosphere grid can create "speckling":  /wiki/spaces/NGDNA/pages/3597565953

All mapping files should be tested as described in the "Testing and Validating Mapping Files" section below.   This testing is necessary, but not sufficient to ensure good results.  

Recommended Settings for E3SM

E3SM v3.LR Mapping table 

state fluxes both

E3SM v3.NARRM Mapping table (tri-grid)

state fluxes both

E3SMv3 with pg2

Similar to E3SMv2 with pg2 but respecting the conventions in Mapping file algorithms and naming convention and using TempestRemap (or mbtempest) for bilinear maps.

1. fluxes:  
   1. if source grid resolution uniformly  >> target grid resolution:  TR aave 
   2. otherwise:    FV2+CAAS 
2. state and vector maps:  
   1. ocn→atm: The state map's non-zero pattern must be a superset of the ocn→atm flux map.  
      1. ocn resolution uniformly >> atm resolution:  TR aave 
      2. otherwise:   FV2+CAAS   or intbilin
      3. avoid bilin due to ocean/land mask consistency issues
   2. lnd↔atm and atm→ocn
      1. if source grid resolution uniformly >> target grid resolution (fine-to-coarse):  TR  aave or intbilin
      2. if source grid resolution uniformly << target grid resolution (coarse-to-fine):  TR bilin or intbilin  (bilin slightly faster)
      3. otherwise:   TR intbilin or FV2+CAAS  (intbilin faster)
      4. For lnd→atm: avoid bilin due to ocean/land mask consistency issues (i.e. 
   3. riv → ocn
      1. continue to use custom maps generated by E3SM/cime/tools/mapping/gen_mapping_files/runoff_to_ocn

"FV2+CAAS" refers to the nonlinear approach from /wiki/spaces/COM/pages/3771301899 where an aave and 2nd order FV map are used with Clip-And-Assured-Sum (CAAS) to provide a monotone conservative map that avoids the blocky artifacts of aave maps and is more accurate than aave maps.  FV2+CAAS is comparable in accuracy to intbilin, adds conservation, but is more expensive.  Thus at present we only use it when necessary for accuracy.  For simplicity (but extra cost) one could consider using FV2+CAAS for all maps.   

Map File Naming Convention

See Mapping file algorithms and naming convention page.  

Testing and Validating Mapping Files

NCO as of 4.9.0-beta01 can test maps for conservation and consistency.   See:  Assess and Address Regridding Weights in Map-Files with ncks.  To check the quality of the mapping file with ncks, use:  "ncks --chk_map map.nc" and then verify:

• sum of area weights is close to physical area 
• mapping weights are between 0 and 1. 
  • For higher order maps, it may be acceptable for small overshoots, but we need to test this. 
  • FV2 map weights will violate this condition, but they are then corrected with CAAS.  
• Consistency:  Row sum of weights (measures what happens when mapping a constant function):  
  • Should be 1 to machine precision for all grids where a >= b.  (global → regional)
  • For grids with a<b (such as ocn→atm) row sums should be 1.0 at all ocean points, 0 at all land points, and between 0 and 1 at coastal points.   
• Conservation:  area_b weighted column sum of weights =  area_a (exact for conservation, close otherwise)  
  • Should be equal to machine precision for all grids where a <= b. (regional → global)
  • For grids with a>b (such as atm->ocn) should be equal to machine precision at all ocean points, 0 at all land points and between 0 and 1 at coastal points
  • Non-conservative maps are expected to have small errors in the above metric.   The traditional ESMF bilinear map does not include areas in the map file and this metric might not be accurate.  

Map file analysis:

There is a python package that can perform a more detailed analysis of mapping files (see  Map file analysis).   It can compute the following additional checks:  

• Row sums:  For o2a maps, separates row sums into those over atm cells that are 100%  ocean (all row sums should be 1), and partial ocean cells (row sums should range from 0 to 1) 
• Column sums: For a2o grids: separate column sums into those over atm cells that are 100% ocean ( should be 1 for conservative maps) and partial ocean cells (row sums should range from 0 to 1 for conservative maps)
• For atmosphere target grids, check if the map is consistent with the atmosphere land fraction mask defined by the flux map (i.e. the map must produce land or ocean data for atm cells that have land or ocean fractions>0)  
• Measure mapping error using a standard test function, plotting the results and computing L2 and max errors.  For atmosphere target grids, separates errors into those over atm cells that are 100% land or ocean, and partial land/ocean cells (which can have much larger errors).     These errors can be compared to know working configurations to determine if they are sufficiently small. 

Older information from v1 and v2 development:

E3SMv2 with pg2

ATM FV grids with physics on the 2x2 physgrid (i.e. pg2) coupled to FV ocean and FV land. E3SMv2 will still use the spectral element dynamical core on the GLL grid, but the coupling between component always happens between the physics columns, which are now on a FV grid. 

fluxes:  FV→FV mono

state and vector maps:  

1. ocn→atm: currently this map must be the same as used for fluxes (FV→FV mono)
   1. This restriction is because the range of the state and vector maps must match the range of the flux map.  
2. lnd↔atm and atm→ocn
   1. if source grid resolution >> target grid resolution (fine-to-coarse):  "FV→FV mono"  
   2. if source grid resolution <= target grid resolution (coarse-to-fine):  use ESMF's bilin.
   3. New as of 2022/10:  TR's FV→FV intbilin should work well for all resolution combinations, replacing both maps above.  Not yet tested in E3SM.  

E3SMv1

ATM SE grids with physics on the "np4" grid (used in V1), coupled to FV ocean and FV land. E3SMv1 uses the same spectral element GLL grid for both dynamics and physics in the atmosphere, so coupling between components happens on this grid.

1. SE→FV  for fluxes:  use "SE→FV mono" 
2. SE→FV for state and vector maps:  SE->FV intbilin
3. FV→SE for fluxes:  FV→SE monotr
   1. The best map is the transpose of the SE->FV map computed above:
   2. This is the most important map, as it is used to compute the domain files and defines the ocean/land mask on the atm/land grid.  
4. FV→SE for state and vector maps:  
   1. ocn→atm: currently this map must be the same as used for fluxes (FV→SE monotr) 
   2. lnd→atm:   if land grid resolution >> atmosphere resolution (fine-to-coarse):  "FV→SE mono". 
   3. lnd→atm:   if land grid resolution <= atmosphere resolution (coarse-to-fine):   ESMF's bilin.  (replace by FV→SE intbilin if it becomes available)

Mapping Analysis and Input Data (lat-lon grids)

For ESMF maps, we previously recommend bilinear maps for coarse-to-fine and aave for fine-to-coarse.  All the TempestRemap algorithms recommended here should work well for either coarse-to-fine or fine-to-coarse.  

SE → Lat/Lon:   "intbilin"

1. "intbilin" if exact conservation is not needed.
2. "mono" if exact conservation is needed and 1st order accurate is ok (i.e. for fine-to-coarse maps).    

Lat/Lon → SE 

1. FV→SE "mono".   Disabling the conservation constraint with --noconserve does not improve accuracy

SE → SE

1. SE→SE "mono"

Motivation for Moving to TempestRemap during development of E3SMv1:

Most mapping files used in the E3SMv0 coupled system and used to remap to lat/lon grids for analysis are based on the ESMF_Regrid package. The ESMF remap algorithms and underlying tools are quite good, but they are targeted at finite volume grids and assume cell centered data.   The TR algorithms have native support for vertex centered finite element grids, such as used by the E3SM atmosphere dycore.

Issues with the current algorithms supported by the ESMF tools: 

Some compsets use ESMF bilinear maps for state variables The bilinear algorithm introduces aliasing artifacts for fine-to-coarse maps.  Some v1 compsets already use TR high-order for atm2ocn maps, which avoid this issue.    

We currently use ESMF conservative maps for fluxes. This algorithm uses piecewise-constant finite volume reconstruction. These maps are good for fine-to-coarse mapping, but will produce blocky artifacts with coarse-to-fine maps.   

ESMF conservative maps for spectral element grids requires us to first generate a "dual grid" that puts a finite volume around each spectral element node.  Constructing this dual grid requires running a Matlab tool that uses Newton iteration to find the dual grid. For a high-resolution grid, this procedure can take several days.  (For information on the dual grid, see SE Atmosphere Grid Overview (EAM & CAM)).  The dual grid is contained in the SCRIP format metadata file.  

TR algorithms natively support finite element grids, do not require the SCRIP dual grid, and give more accurate results when remapping to/from a spectral element atmosphere grid.

TR algorithm recommended for state variables is cell integration based, so can always be used when mapping from high-res to low-res grids.

Inline mapping: TempestRemap algorithms are part of the MOAB coupler, making it possible to eliminate mapping files and have them computed as needed.

Land/Ocean Mask consistency:   The flux ocn2atm map is used to define the land/ocean mask on the atmosphere grid. All other ocn2atm maps (state and vectors) must have the same range as the map uses for fluxes.  That is, if a point on the atmosphere grid receives ocean data from the flux map, it must also receive ocean data from all other maps.  The aave and bilinear maps do not have the same range, and thus if an aave map is used for fluxes, it must be used for all other ocn2atm maps.   We speculate that TR maps all have the same range and thus we can use high order maps for ocn2atm state and vectors.   

We had some issues with TempestRemap, where the initial land mask was not very good.  These were resolved here: /wiki/spaces/CMDV/pages/128827525

Notation and Abbreviations

SE =  Atmosphere spectral element dycore mesh ( SE Atmosphere Grid Overview (EAM & CAM) ). Usually in "np4" configuration where each element has a 4x4 array of GLL nodes.   

GLL = stands for "Gauss-Lobatto-Legende" quadrature, but we often use it to refer to the quadrature "nodes" that define the SE grid. Although quantities are technically continuous and differentiable on the SE grid, the discrete values live at the GLL nodes, which should be thought of as points rather than volumes or areas. 

FV =  MPAS finite volume mesh, the FV r05 and r0125 land and river runoff grids, and SE atmosphere "physgrids"  pg2 and pg3 options.     

Grid description/template files: 

$atmgrid = SE Atmosphere grid in Exodus format. These are .g files. 
$ocngrid = FV MPAS grid file in "scrip" format. These are .scrip.nc files

These grid description files are available on the E3SM inputdata server E3SM Input Data Repository.  The atmosphere grids are in collected in the mapping folder:   https://web.lcrc.anl.gov/public/e3sm/mapping/grids/. The MPAS grids are together with the inputdata for each grid.  For example:  https://web.lcrc.anl.gov/public/e3sm/inputdata/ocn/mpas-o/oEC60to30/

Note 1: For Atmosphere RRM grids, the Exodus .g file should be put in https://web.lcrc.anl.gov/public/e3sm/mapping/grids/.  But cubed-sphere grids are computed internally by the model and we dont have .g files for these grids.  They can be computed via a utility included with TempestRemap:

./GenerateCSMesh --alt --res 30  --file NE30.g                                               # the standard atmosphere SE grid, using the np4" GLL spectral element
./GenerateVolumetricMesh --in NE30.g --np 2 --out NE30pg2.g --uniform     # An atmosphere "FV" grid (for the new pg2 option where physics is on a FV grid)

Note 2: It is important to use the "–alt" option with older versions of TR, so the mesh ordering matches what is used internally by E3SM.  This option was made the default in 2020/2.  

Note 3:  For SE atmosphere grids, the Exodus file is what is used by the model, and it contains the corners of all the spectral elements.  The dual grid (SCRIP format) metadata files are not needed by TempestRemap.  

Using TempestRemap

Generic TR Commands to Generate Mapping Files

Update to TR options 2020/2/21:  to fix precision issues, with row sums and conservation errors not close to machine precision at high resolution meshes, added the "–correct_areas" option to the recommendations.  

2022/9 update:  newer versions of TR replace the --mono* options with --method mono*

E3SM v2 (all maps are FV->FV)

Bilinear maps:

• With E3SMv2's transition to PG2, we now often need bilinear maps for state variables.  These currently are not supported by TR and need to be computed with the ESGF utility
• ESGF options for bilinear maps:  ADD HERE.  

FV→FV aave (conservative, monotone, 1st order)

• The classic cell integrated piecewise constant map.  Equivalent to the ESMF "aave" map and NCO's built in map.  
• ./GenerateOfflineMap --in_mesh $ocngrid --out_mesh $atmgrid --ov_mesh overlap_mesh.g --in_type fv --in_np 1 --out_type fv --out_np 1 --correct_areas  --out_map map_ocn2atm_mono.nc

FV→FV integrated bilinear

• promising new capability in TR 2022/10, not yet tested in E3SM

E3SM v1 (maps with SE grids.  also sometimes used to produce E3SM v2 initial conditions which are on SE grids)

SE→SE mono (conservative, monotone, 1st order)

• ./GenerateOfflineMap --in_mesh $atmgrid --out_mesh $atmgrid2 --ov_mesh overlap_mesh.g --in_type cgll --in_np 4  --out_type cgll --out_np 4 --mono --correct_areas --out_map map_atmgrid_to_atmgrid2_mono.nc

SE→FV mono (conservative, monotone, 1st order)

• ./GenerateOfflineMap --in_mesh $atmgrid --out_mesh $ocngrid --ov_mesh overlap_mesh.g --in_type cgll --in_np 4  --out_type fv  --mono --correct_areas --out_map map_atm2ocn_mono.nc

SE→FV intbilin (nonconservative, monotone, higher order)

• ./GenerateOfflineMap --in_mesh $atmgrid --out_mesh $ocngrid --ov_mesh overlap_mesh.g --in_type cgll --in_np 4  --out_type fv --mono3 --noconserve --correct_areas --out_map map_atm2ocn_intbilin.nc

SE→FV highorder. (conservative, nonmonotone)

• ./GenerateOfflineMap --in_mesh $atmgrid --out_mesh $ocngrid --ov_mesh overlap_mesh.g --in_type cgll --in_np 4  --out_type fv --correct_areas --out_map map_atm2ocn_highorder.nc

FV→SE monotr (conservative, monotone, 1st order)

• The transpose of the SE→FV mono map: 
• ./GenerateTransposeMap --in map_atm2ocn_mono.nc --out map_ocn2atm_monotr.nc

FV→SE highorder (conservative, non-monotone)

• ./GenerateOfflineMap --in_mesh $ocngrid --out_mesh $atmgrid --ov_mesh overlap_mesh.g --in_type fv --in_np 2 --out_type cgll --out_np 4  --out_map map_ocn2atm_highorder.nc
• Do these maps need "–correct_areas" ?

FV→SE mono (conservative, monotone, 1st order)

• ./GenerateOfflineMap --in_mesh $ocngrid --out_mesh $atmgrid --ov_mesh overlap_mesh.g --in_type fv --in_np 1 --out_type cgll --out_np 4 --out_map map_ocn2atm_mono.nc
• Do these maps need "–correct_areas" ?

FV→SE intbilintr  (nonconservative, nonmonotone, higher order)

• The transpose of the SE→FV intbilin map.  No negative weights, but weights can be larger than 1.   
• ./GenerateTransposeMap --in map_atm2ocn_intbilin.nc --out map_ocn2atm_intbilintr.nc

The Overlap Mesh

IMPORTANT!!!: MAPS grids often have holes where there are no mesh points. When computing the TempestRemap overlap grid (step 1 below), the grid with less coverage must be the "a" grid, while the global grid should be the "b" grid.   To generate the overlap grid:

./GenerateOverlapMesh --a $ocngrid --b $atmgrid --out overlap_mesh.g

TR support in NCO and ncremap

The TR maps discussed in "Recommended Settings" above include many options and requirements (transposes, ocean grid first) that might slow-down casual TR users. NCO's ncremap provides a "Make Weight Files" (MWF) mode that automatically generates and correctly names all recommended TR (or ESMF) maps. Instructions and examples for MWF mode are summarized on the ncremap page here, with more in-depth discussion in the NCO manual here.

(This page was called Transition to TempestRemap for Atmosphere grids for several years, so this sentence is to ensure that string still appears)