The purpose of this page is to document the procedure for adding support for new atmosphere grids. The process should be the same for new uniform resolutions as well as for new regionally-refined meshes, although some settings will need to be changed for new regionally-refined mesh configurations. This page is a work in progress, and will be updated as this process is refined and (eventually) made more automated. This documentation is an update of a document written by @Mark Taylor and Colin Zarzycki, available as a Google Doc here.

Table of Contents

Child Pages

Useful Example Pages:

• https://acme-climate.atlassian.net/wiki/spaces/NGDNA/pages/3086549115

•  NE512 and NE1024 configurations

•  

Archives of already-created atmosphere grid and mapping files can be accessed at https://web.lcrc.anl.gov/public/e3sm/mapping/grids/ and https://web.lcrc.anl.gov/public/e3sm/mapping/maps/ . This page focuses on creating new versions of these files.

For the purpose of this step-by-step guide, we will walk through each step using an example of regenerating the ne4 grid that is already currently supported. For a new grid (i.e., ne512), just replace ne4 with the desired grid identifier. This is determined in Step 1 Generate a new "grid" file below, and the rest should follow.

For this example, I will assume a few environment variables, which we should set now. First, determine where to put the generated output files. For big grids, this can take up quite a bit of space, so choose a location with sufficient space. I have a symlink to my CSCRATCH directory on NESRC in my home directory, so I will put grid files there for this example:

Types of Atmosphere grid metadata files

See SE Atmosphere Grid Overview (EAM & CAM) for description of the spectral elements, GLL nodes, subcell grid and dual grid.   

• Exodus file: "ne4.g".   This is a netcdf file following Exodus conventions.  It gives the corners of all elements on the sphere and their connectivity.  It is independent of the polynomial order used inside the element ("np").  

  • This file is used by TempestRemap (TR) to generate mapping files.  The polynomial order is a command line option and the GLL nodes are internally generated by TR.  

• SCRIP file:  "ne4pg2.scrip.nc".   This file contains a description of the atmosphere physics grid n the format used by the original incremental remap tool SCRIP. It is used for most output and also used to generate mapping files between components and for post-processing of most output.

• Less common “GLL” metadata files needed for specialized purposes:

  • SCRIP file:  "ne4np4_scrip.nc".   This file contains a description (SCRIP format) of the GLL dual grid. It includes the locations of the GLL nodes and artificial bounding polygons around those nodes.   Ideally the spherical area of each polygon will match the GLL weight ("exact GLL areas"), but not all tools can achieve exact areas.  Inexact areas does not impact the accuracy of the resulting mapping algorithm, it just means that mass will not be exactly conserved by the mapping algorithm.  

    • Needed in E3SMv2 configurations by the cube_to_target topography dataset generation tool.

    • Historically used by E3SMv1 to create ocean/atm mapping files (exact GLL areas are important here, to conserve fluxes)

    • Historically used by E3SMv1 for regional and sub-gridscale regridding post-processing tools ( https://acme-climate.atlassian.net/wiki/spaces/SIM/pages/754286611/Regridding+E3SM+Data+with+ncremap, http://nco.sf.net/nco.html#sgs)

  • latlon file:  "ne4np4_latlon.nc".   This file contains a list of all the GLL nodes in the mesh (in latitude/longitude coordinates).   The list of GLL nodes must be in the the internal HOMME global id ordering, matching the ordering used in CAM and EAM native grid output.   It also contains the connectivity of the GLL subcell grid.   

    • This file is used by CAM's interpic_new utility, and graphics programs Paraview and Visit when plotting native grid GLL output.

Step-by-step guide

1. Generate a new atmosphere "grid" file

Requirements: TempestRemap

In order to generate mapping files to do interpolation between a new atmosphere grid and the other component grids, we need to create a file that describes the atmosphere grid representation. The TempestRemap tool (https://github.com/ClimateGlobalChange/tempestremap) can be used to recreate the internal representation of the spectral element grids used in HOMME. The grid mesh file will be saved in an "exodus" type file (with a .g extension). 

TempestRemap can now be pulled in with NCO via conda, or can be built from source from the Github repository. To build from source, make sure a netCDF module is loaded. This can be accomplished by sourcing an env_mach_specific.sh from a working case on the target machine before building. Then, 

This will build TempestRemap into tempestremap/bin. For the rest of the tutorial, assume that the environment variable ${tempest_root} points to this directory (or where a conda-build of TempestRemap exists):

We can generate an ne4 mesh easily now:

For RRM grids, SQuadGen should be used (available in a separate repository on Github here). See Generate the Grid Mesh (Exodus) File for a new Regionally-Refined Grid for details and examples. The example above follows the convention ne<res>.g. RRM meshes should follow the convention <area refined>_<base resolution>x<refinement level>.g. For example, for a RRM with 4x refinement from ne30 to ne120 over CONUS, we should use the convention conus_ne30x4.g (note that existing meshes use the naming convention <area of refinement>x<refinement level>v<version>.g, but future meshes should use the new naming convention; we also plan to eventually change the extension to _exodus.nc, but I believe this will break nco functionality to automatically detect when to use TempestRemap for mapping when detecting exodus files).

The Exodus file contains only information about the position of the spectral element on the sphere. For SE aware utilities such as TempestRemap, they can use the polynomial order and the reference element map to fill in necessary data such as the locations of the nodal GLL points. For non-SE aware utilities, we need additional meta data, described in the next section.   

2A. Generate control volume mesh files for E3SM v2 "pg2" grids 

Requirements:

• exodus mesh file

• TempestRemap

In E3SM v2, we switched to running physics on a FV pg2 grid and mapping files between the atmosphere and other components will be FV to FV type maps using the pg2 grids.  These can be generated by TempestRemap directly from the exodus mesh file:

2B. Generate "dual grid" mesh files (SCRIP and lat/lon format) for E3SM v1 "np4" GLL grids

Requirements:

• exodus mesh file

• homme_tool

Note: in E3SM v2, we switched to running physics on a FV pg2 grid and mapping files between the atmosphere and other components will be FV to FV type maps using the pg2 grids.   The spectral element "np4" grid is still used internally by the dynamics and for initial conditions, so the metadata described in this section is still needed for some analysis and preparation of initial conditions.   

Mapping files used by the coupler for mapping fluxes between SE "np4" and FV grids should be generated with TempestRemap and only need the exodus grid description (which provides the locations of the corners of the quadrilaterals that form the elements of the cube-sphere mesh) generated in Step 1 above. However, a handful of pre- and post-processing tools require a finite-volume equivalent of the spectral element grid (these tools include the surface roughness calculation in the land tool cube_to_target, ESMF mapping files used for interpolating land surface data to target grid, and post-processing regional and subgrid remapping tools). We refer to this finite-volume description of the SE grid as the "dual grid" to the SE grid (see page describing atmosphere grids in more detail here).     

Generate SCRIP “dual grid” with homme_tool:

1. Be sure your environment matches the software environment loaded by E3SM by executing the output of this command:   e3sm/cime/scripts/Tools/get_case_env

2. Use cmake to configure and compile standalone HOMME.  On a supported platform with the CIME environement, this should work out-of-the-box.  See e3sm/components/homme/README.cmake

3. compile the HOMME tool utility: 

   1. cd /path/to/workingdir 

   2. make -j4 homme_tool

   3. executable:   /path/to/workingdir/src/tool/homme_tool

4. Edit e3sm/components/homme/test/tool/namelist/template.nl and specify the grid resolution or RRM file

   1. For ne512, this would be ne = 512. For RRM grids, leave ne = 0, but will need to edit where the exodus grid file comes from

   2. for non-RRM grids using the older E3SM v1 dycore, add cubed_sphere_map=0 to template.nl

5. See e3sm/components/homme/test/tool/test.job for examples of how to run the script and then use an NCL utilities to process the tool output into SCRIP and latlon formats.  

Specific details for running at NERSC on Cori(knl):

1. Create a batch script hange "account" in the sbatch directives at the top of the script. For example, set #SBATCH --account=e3sm

2. cmake -C /path/to/e3sm/components/homme/cmake/machineFiles/cori-knl.cmake  -DPREQX_NP=4 /path/to/workingdir

3. Make sure a working NCL is in your PATH. On Cori, add the following to the script: module load ncl.

2C. Atmospheric mesh quality

Atmospheric RRM mesh quality can be measured with the “Max Dinv-based element distortion” metric. This will be printed in the log file for standalone HOMME or EAM simulations (and can be obtained from the log files during the topo generation step). It measures how distorted the elements become in the mesh transition region. It is the ratio of the two singular values of the 2x2 derivative matrix of the element map to the unit square, representing the ration of the largest length scale to the smallest length scale. A grid of perfect quadrilaterals will have a value of 1.0. The equal-angle cubed-sphere grid has a value of 1.7.   A high quality regionally refined grid will have a value less than 4. With a high quality grid, usually one can run with the timesteps used in a uniform grid with matching fine resolution. RRM grids with a value > 4 may require smaller timesteps for stability. Very large values indicate a problem with the grid and it should be redesigned.

3. Generate mapping files

Requirements:

• TempestRemap

• ESMF_RegridWeightGen

• ncremap

• grid descriptor files for each component that exists on a different grid
  (atmosphere, ocean, possibly land if on a different grid than the atmosphere)

In order to pass data between different components at runtime, a set of mapping files between each component is generated offline. These mapping files will also be used in Step 4 below (generating domain files).

See Transition to TempestRemap for Atmosphere grids for a discussion of different remap algorithms and when to use each.

TempestRemap and ESMF are the backends that generate the mapping weights, but this is all nicely encapsulated using ncremap. Tempest is the preferred method for creating mapping files.     The ncremap function, calling TempestRemap or ESMF, will decide which of these two tools to use based on the atmospheric input file.  If the *.scrip file was used, then ESMF will be called.  If the *.g file was used, then TempestRemap will be called.   The ESMF tools are adequate for making atmosphere-only-type component sets for E3SM, but this tool is less conservative than TempestRemap.   If you are making grids for a coupled run, then TempestRemap should be used.  

The easiest way to make sure ncremap is up to date and will work for this workflow, we can source the ESMF-Unified conda environment. Instructions for supported machines can be found here. For Perlmutter, it is as simple as:

For unsupported machines you may need to install the conda environment via:

TempestRemap was built in Step #1 above.     We need to tell ncremap where to find our TempestRemap binaries, so we need to append to our PATH:

And now we can use ncremap to generate ALL the needed mapping files between two grids, in this example the ne4 atmosphere and the oQU240 ocean grid (for the moment, we will put the land on the same grid as the atmosphere):

In the above code, the "-P mwf" option triggers a specific procedure that invokes multiple ncremap commands which each invoke multiple Tempest commands.It produces 3 mapping files each for a2o and o2a. This multitude of mapping files for each component→component case are needed because fluxes need conservative mapping and states need non-conservative mapping. Because "-P mwf" results in a great deal of output being flushed to screen, it is mostly suppressed by default. To see all the output or to figure out how to run all the nco and tempest remap commands one-by-one, add "--dbg_lvl=2" to the ncremap command. This will print the commands but not execute them. The user can then use one of the printed commands, again with "--dbg_lvl=2" to see the actual commands being sent to Tempest.

If this command is successful, it should produce many mapping files in ${output_root}, that look something like

where ${ocn_grid}, ${atm_grid}, and ${lnd_grid} are the names of the ocean and atmos grid provided above in the --nm_src and --nm_dst arguments, and ${datestring} is the date string provided above in the --dt_sng argument to ncremap.   The ${method} can be monotr, highorder, mono, intbilin, aave, blin, ndtos, nstod, and patc.  (TODO:  What do all these mean?)   If using the tri-grid option, the land grid files will be created from the ${lnd_grd}.  

For pg2 grids, the ncremap invocations are as follows, using ne30pg2 atmosphere, oEC60to30v3 ocean, and half-degree land grids as examples.

4. Generate domain files

Domain files are needed by the coupler and the land model at runtime.    The land model uses the mask to determine where to run and the coupler use the land fraction to merge fluxes from multiple surface types to the atmosphere above them.  Domain files are created from the mapping files created in the previous step, using a tool provided with CIME in ${e3sm_root}/cime/tools/mapping/gen_domain_files. This directory contains the source code for the tool (in Fortran 90) and a Makefile.    Cloning E3SM is now required to obtain code within the distribution.   To clone E3SM, 

Assuming ${e3sm_root} is set and the location of TempestRemap binaries is added to ${PATH}, the following script should generate the appropriate domain files:

NOTE - on Perlmutter the “OS” environment variable is not set, so to work around this simply set “OS=LINUX“
If this command is successful, it should produce many domain files in ${output_root}/domain_file that look something like

5. Generate topography file 

Generating the topography and related surface roughness data sets is a detailed process that has been moved to it’s own page, with detailed instructions depending on the model version (V1, V2, V3)