Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 58 Next »

DRAFT  

This document recommends switching many of the E3SM mapping files to those generated with the TempestRemap (TR) algorithms for all grids involving atmosphere data.

These should be used for all V2 configurations.  This will also ease the transition to inline mapping in V2 with the MOAB based coupler which has inline support for Tempest algorithms.  

Motivation:

Most mapping files used in the E3SM coupled system and used to remap to lat/lon grids for analysis are based on the ESMF_Regrid package. The ESMF remap algorithms and underlying tools are quite good, but they are targetted at finite volume grids and assume cell centered data.   The TR algorithms have native support for vertex centered finite element grids, such as used by the E3SM atmosphere dycore.

Issues with the current algorithms supported by the ESMF tools: 

  1. Some compsets use ESMF bilinear maps for state variables The bilinear algorithm introduces aliasing artificats when mapping from a high-resolution to low-resolution region (downscaling).  Some v1 compsets already use TR highorder for atm2ocn maps, which avoid this issue.    
  2. We currently use ESMF conservative maps for fluxes. This algorithm uses piecewise-constant finite volume reconstruction. These maps are good for downscaling, but will produce blocky artifacts when upscaling. 
  3. ESMF conservative maps for spectral element grids requires us to first generate a "dual grid" that puts a finite volume around each spectral element node.  Constructing this dual grid requires running a Matlab tool that uses Newton iteration to find the dual grid. For a high-resolution grid, this procedure can take several days.  (For information on the dual grid, see CAM-SE Grid Overview).  The dual grid is contained in the SCRIP format metadata file.  
  4. TR algorithms natively support finite element grids, do not require the SCRIP dual grid, and give more accurate results when remapping to/from a spectral element atmosphere grid.
  5. TR algorithm recommended for state variables is automatically downscaling, so can always be used when mapping from high-res to low-res grids.
  6. Inline mapping: TempestRemap algorithms are part of the MOAB coupler, making it possible to eliminate mapping files and have them computed as needed.
  7. Land/Ocean Mask consistency:   The flux ocn2atm map is used to define the land/ocean mask on the atmosphere grid. All other ocn2atm maps (state and vectors) must have the same range as the map uses for fluxes.  That is, if a point on the atmosphere grid receives ocean data from the flux map, it must also receive ocean data from all other maps.  The aave and bilinear maps do not have the same range, and thus if an aave map is used for fluxes, it must be used for all other ocn2atm maps.   We speculate that TR maps all have the same range and thus we can use high order maps for ocn2atm state and vectors.   


Naming conventions and Notation:  

SE =  Atmosphere spectral element dycore mesh ( CAM-SE Grid Overview )

FV =  MPAS finite volume mesh or, for the v2 trigrid setups, the FV r05 and r0125 land and river runoff grids. 

Grid description/tempalte files: 

$atmgrid = SE Atmosphere grid in Exodus format. These are .g files. 
$atmw = SE Atmosphere grid weights. These are _latlon.nc files
$ocngrid = FV MPAS grid file in "scrip" format. These are .scrip.nc files

These grid description files are available on the ACME inputdata server /wiki/spaces/WORKFLOW/pages/7998629.  The atmosphere grids are in collected in the mapping folder:   https://acme-svn2.ornl.gov/acme-repo/acme/mapping/grids/.    The MPAS grids are together with the inputdata for each grid.  For example:  https://acme-svn2.ornl.gov/acme-repo/acme/inputdata/ocn/mpas-o/oEC60to30/

Note 1: For Atmosphere RRM grids, the Exodus .g file should be put in https://acme-svn2.ornl.gov/acme-repo/acme/mapping/grids/.  But cubed-sphere grids are computed internally by the model and we dont have .g files for these grids.  They can be computed via a utility included with TempestRemap:

./GenerateCSMesh --alt --res 30  --file NE30.g                                               # the standard atmosphere SE grid, using the np4" GLL spectral element
./GenerateVolumetricMesh --in NE30.g --np 2 --out NE30pg2.g                  # An atmosphere "FV" grid (for the new pg2 option where physics is on a FV grid)

Note 1: It is important to use the "–alt" option, so the mesh ordering matches what is used internally by E3SM.  This option was made the default in 2020/2.  

Note 2:  For SE atmosphere grids, the Exodus file is what is used by the model, and it contains the corners of all the spectral elements.  The dual grid (SCRIP format) metadata files are not needed by TempestRemap.  


Proposed Map file suffix naming convention:  (Edited 2017/11)


_aave.nc:        ESMF conservative/monotone "area averaged" maps
_bilin.nc:       ESMF bilinear
_mono.nc       TempestRemap conservative, monotone map
_highorder.nc TempestRemap high-order, conservative, non-monotone  ( several TR options in this category )
_intbilin.nc       TempestRemap integrated bilinear:   monotone, non-conservative
_monotr.nc     TempestRemap transpose of a mono map.  
_highordertr.nc   TempestRemap transpose of a highorder map.  (not recommended) 


Note: Some of the maps used for state variables in ACME v1 are labeled "_conserve.nc", which are conservative, high-order, non-monotone maps ("highorder" in the above proposed naming convention), which causes some confusion because  "conserve" implies it's a replacement for an "aave" type map, when really that type of map is a replacement for "bilin".   

The Overlap Mesh

IMPORTANT!!!: MAPS grids often have holes where there are no mesh points. When computing the TempestRemap overlap grid (step 1 below), the grid with less coverage
must be the "a" grid, while the global grid should be the "b" grid.   To generate the overlap grid:

./GenerateOverlapMesh --a $ocngrid --b $atmgrid --out overlap_mesh.g


Update to TR options 2020/2/21:  to fix precision issues, with row sums and conservation errors not close to machine precision at high resolution meshes, added the "–correct_areas" option to the recommendations.  


TR options to generate the various maps:

  • SE→FV mono. (conservative, monotone, 1st order)
    • ./GenerateOfflineMap --in_mesh $atmgrid --out_mesh $ocngrid --ov_mesh overlap_mesh.g --in_type cgll --in_np 4  --out_type fv  --mono --correct_areas --out_map map_atm2ocn_mono.nc
  • SE→FV intbilin (nonconservative, monotone, higher order)
    • ./GenerateOfflineMap --in_mesh $atmgrid --out_mesh $ocngrid --ov_mesh overlap_mesh.g --in_type cgll --in_np 4  --out_type fv --mono3 --noconserve --correct_areas --out_map map_atm2ocn_intbilin.nc
  • SE→FV highorder. (conservative, nonmonotone)
    • ./GenerateOfflineMap --in_mesh $atmgrid --out_mesh $ocngrid --ov_mesh overlap_mesh.g --in_type cgll --in_np 4  --out_type fv --correct_areas --out_map map_atm2ocn_highorder.nc
  • FV→SE monotr. (conservative, monotone, 1st order)
  • FV→SE highorder. (conservative, non-monotone)
    • ./GenerateOfflineMap --in_mesh $ocngrid --out_mesh $atmgrid --ov_mesh overlap_mesh.g --in_type fv --in_np 2 --out_type cgll --out_np 4  --volumetric --out_map map_ocn2atm_highorder.nc
    • Does these maps need "–correct_areas" ?
  • FV→SE mono. (conservative, monotone, 1st order)
    • ./GenerateOfflineMap --in_mesh $ocngrid --out_mesh $atmgrid --ov_mesh overlap_mesh.g --in_type fv --in_np 1 --out_type cgll --out_np 4 --volumetric --out_map map_ocn2atm_mono.nc
    • Does these maps need "–correct_areas" ?
  • FV→SE intbilintr  (nonconservative, nonmonotone, higher order)
  • FV→FV mono (conservative, monotone, 1st order)
    • The classic cell integrated piecewise constant map.  Equivalent to the ESMF "aave" map and NCO's built in map.  
    • ./GenerateOfflineMap --in_mesh $ocngrid --out_mesh $atmgrid --ov_mesh overlap_mesh.g --in_type fv --in_np 1 --out_type fv --out_np 1 -correct_areas  --out_map map_ocn2atm_mono.nc
  • FV→FV highorder. (conservative, non-monotone, high order)
    • ./GenerateOfflineMap --in_mesh $atmgrid --out_mesh $ocngrid --ov_mesh overlap_mesh.g  --in_np 2 --in_type fv  --out_type fv  --out_np 1 -correct_areas --out_map map_atm2ocn_highorder.nc
    • --in_np 2 means all the nearest neighbor cells will be used for a higher order reconstruction of the source data, which will then be integrated over the target cell.  Unlike the SE case, no mass matrix is used for an FV target grid, and thus --out_np is ignored (and will always be 1 internally).  Paul Ullrich please confirm!


UPDATE 2020/2/21:  Removing highorder maps which can have negative weights.  It's probably a bad idea to use negative weights in a coupled simulation, and we shouldn't recommend this without detailed evaluation.        

Recommended Settings:   For ATM SE grids coupled to FV ocean and FV land

  1. SE → FV  for fluxes:  use "mono" 
  2. SE → FV for state variables:  "highorder". ). not recommended.  
  3. SE → FV for state variables:  intbilin?
  4. FV → SE for fluxes:  monotr
    1. The best map is the transpose of the SE->FV map computed above:
    2. This is the most important map, as it is used to compute the domain files and defines the ocean/land mask on the atm/land grid.  
  5. FV → SE for state variables and velocity:  highorder.   not recommended.  
    1. Tests with ocean120→NE30, analytic fields, show this map is more accurate than the transpose map. 
  6. FV → SE mono.  
    1. This is also a good monotone/conservative map for fluxes, but not quite as accurate (in analytic tests) as the transpose map recommended above.  
  7. FV → SE for state variables
    1. ocn→atm: currently this map must be the same as used for fluxes.     
    2. lnd→atm:   if land grid resolution >> atmosphere resolution:  "mono"  
    3. lnd→atm:   if land grid resolution <= atmosphere resolution:   ESMF's bilin.  
    4. DESIRED:  a version of intbilin for FV source grids.   


Recommended Settings:   For ATM FV grids (the new "pg2" option)  coupled to FV ocean and FV land

  1. fluxes:  mono
  2. state variables:
    1. ocn→atm: currently this map must be the same as used for fluxes.     
    2. lnd↔atm and atm→ocn
      1. if source grid resolution >> target grid resolution:  "mono"  
      2. source grid resolution <= target grid resoluiton:  ESMF's bilin
    3. DESIRED:  a version of intbilin for FV source grids. 

Recommended Settings:   Atm to/from Lat/Lon maps for analysis and interpolating input data

For ESMF maps, we previously recommend bilinear maps for upscaling and aave for downscaling.  All the TempestRemap algorithms recommended here should work well for either upscaling or downscaling.  The choice depends mostly on the tradeoff between monotonicity and accuracy:

  1. SE → Lat/Lon:   "intbilin"
    1. "intbilin" if exact conservation is not needed.
    2. "mono" if exact conservation is needed and 1st order accurate is ok (i.e. for downscaling).    
  1. Lat/Lon → SE 
    1. FV→SE "mono".   Disabling the conservation constraint with --noconserve does not improve accuracy
    2. FV→SE "highorder" is accurate, but care must be taken with negative weights.  . 
    3. NOTE:  For Lat/Lon grids, check the min/max of the weights - if the values are far outside the "[0,1]" interval, may need more coefficients for Vandermonde inverse near poles: edit LinearRemapFV.cpp: nRequiredFaceSetSize = 2*nCoefficients. 


Testing

Important issues to test:

  • Conservative maps are conservative to machine precision w.r.t. to the internal weights used by the model.   If this is not the case, the E3SM coupler will introduce an area ratio correction.  This corrections is printed by the coupler at run time and can be checked in the log files.   
  • Overlap mesh was generated correctly (with the ocean grid first).  If not, all the maps will still work, but there will be severe accuracy issues near coastlines
  • Accuracy:  TempestRemap can automatically generate some standard test functions which can then be remapped and we can compute the mapping error, and check if it is at the expected level.  

NEW AS OF NCO 4.9.0-beta01 and TR as of 2019/11:  Check the quality of the mapping file with "ncks --chk_map map.nc"

  1. sum of area weights is close to physical area 

  2. mapping weights are between 0 and 1.  for higher order maps, it may be acceptable for small overshoots, but we need to test this. 

  3. Consistency:  Row sum of weights = 1.

    1. Should be 1 to machine precision for all grids where a >= b.  (global → regional)
    2. For grids with a<b (such as ocn→atm) row sums should be 1.0 at all ocean points, 0 at all land points, and between 0 and 1 at coastal points.   
  4. Conservation:  area_b weighted column sum of weights =  area_a (exact for conservation, close otherwise)  
    1. Should be equal to machine precision for all grids where a <= b. (regional → global)
    2. For grids with a>b (such as atm->ocn) should be equal to machine precision at all ocean points, 0 at all land points and between 0 and 1 at coastal points
    3. Non-conservatives maps are expected to have small errors.   
  5. no missing target points (unless source grid has holes)


NOTE:   The land mask, as used by the land model is computed by remaping the ocean mask (on the ocean grid) to the atmosphere grid.  On the ocean grid, the ocean/land mask is all 0's or 1's  (that is, every point is either ocean or land).  Actually, on an MPAS grid, the land points are all removed, so the mask is all ocean.   This field is then mapped on to the land grid (when creating domain files).  This results in a land mask on the atmosphere/land grid with fractional land points ( 0< mask<1) near coastlines.  

We had some issues with TempestRemap, where the initial land mask was not very good.  These were resolved here: /wiki/spaces/CMDV/pages/128827525

TR support in NCO and ncremap

The TR maps discussed in "Recommended Settings" above include many options and requirements (transposes, ocean grid first) that might slow-down casual TR users. NCO's ncremap provides a "Make Weight Files" (MWF) mode that automatically generates and correctly names all recommended TR (or ESMF) maps. Instructions and examples for MWF mode are summarized on the ncremap page here, with more in-depth discussion in the NCO manual here.

  • No labels