CAM-SE works with data stored at Gauss-Lobatto-Legendre (GLL) nodes. These nodes can be interpreted in several ways shown in various grids below. In the "SE native grid", "SE subcell grid" and "SE dual grid", the GLL nodes are the same (the green dots in the figure). Internally, the dycore uses the CAM-SE native grid connectivity, but the rest of CAM and E3SM has no knowledge of the grid connectivity and only sees data at GLL nodes and their associated area weight. The other other types of connectivity are not used within the model, but are sometimes used by pre- and post-processing tools.

E3SM v2 adds the option to run the physics on a "physgrid", also shown below. The physgrid is a conventional finite volume (FV) grid, where each physics column is located in the center of the FV cell and represents the cell averaged quantities. The physgrid gives E3SM the flexibility to run the physics at lower or higher resolution than the dynamics.

Most native grid atmosphere output for E3SM V2 will be on the pg2 physgrid. Native grid atmosphere output from E3SM V1 is on the np4 GLL grid.

CAM-SE Native Grid (GLL data)

Here is a low-resolution CAM-SE grid. This is a "ne4np4" grid, meaning that it has 4x4 spectral elements in each cube face (the "ne" value), and each spectral element has a grid of 4x4 (the "np" value) Gauss-Lobatto points inside it. The blue lines show the edges of the spectral elements. CAM-SE is collocated, meaning that all variables (U,V,T, etc...) are carried around on the GLL nodes. CAM physics is computed in columns at GLL nodes. Each column has a Gauss-Lobatto weight associated with it, which can also be interpreted as an area associated with each node. The sum of these weights is 4pi (the area of the unit sphere). One can see from the figure that GLL nodes are not quite equally spaced - they cluster at the edges of the spectral elements.

Give any SE grid with N elements, the number of unique GLL nodes will be N*(np-1)^2 + 2

To map data between SE native grids and FV grids, TempestRemap should be used (see Recommended Mapping Procedures for E3SM Atmosphere Grids ). SE GLL data is not naturally represented as FV type cell averages, and thus mapping tools which only work with cell centered data (ESMF, SCRIP) should be avoided.

The PG2 "physgrid"

In E3SM v2, we run the physics on a finite volume grid, where each spectral element cell is further divide into subcells. PG2 computes the physics on a 2x2 FV grid within each spectral element (shown below). On the physgrid, the physics columns are located at the cell center (green dots below), but should be considered cell average values over each FV cell, rather than nodal values at the green points.

For v2 simulations, most output will be on the PG2 physgrid.

Give a SE grid with N elements, the number of unique physgrid nodes will be N*(pg)^2

The physgrid is a FV grid and can be used with any tool that works with cell centered FV data (such as ESMF map file algorithms).

SE "subcell" grid (GLL data)

Most analysis tools and other tools cannot handle higher order elements such as used in CAM-SE. For these codes, we produce metadata that divide each spectral element into subcells. This subcell grid is shown below. The GLL nodes are the vertices of the subcell grid. This meta data can be used by Paraview and Visit to plot native grid CAM-SE output. The metadata is stored in a grid template file that typically has a file name of the form "<gridname>_latlon_<date>.nc" in the grids directory of the E3SM inputdata server. From Euler's polyhedron formula, we know if there are N physics columns (GLL nodes), there will be N-2 subcells (all quads) and N/2-1 subcell edges.

SE "dual" grid (GLL data)

Some applications require treating the GLL nodal data as FV cell centered data. To do this, we need to construct the "dual grid" of the GLL subcell grid, resulting in a grid where each polygon contains a single GLL nodal point. This is not recommended and as one can see below is an unnatural representation of the GLL nodal data.

Constructing a SE dual grid is non-trivial and we have explored several options:

For conservation, the areas of these polygons must match the weight of each GLL node. If we construct this dual grid in the usual way, by connecting the centers of all the subcells shown in the subcell grid, the areas of the cells will in general not match the GLL weights. We thus have to perform an iteration, similar to spring dynamics, tweaking the polygons until the areas are correct. We have several different algorithms to do this (examples shown below). The resulting polygons can be a little odd. First plot below (chevrons): we allow pentagons and hexagons so that the algorithm will converge faster and to more uniform cells, but this means that some of them will be slightly non-convex as some of the hexagons turn into chevrons. ESMF can handle non-convex cells, but other utilities may require they be convex. Second plot below (natural): If we only allow dual grid vertices at cell centers of the subcell grid, we get convex polygons, but they are less regular. Third plot below (pentagons): The dual grid files currently used for cubed-sphere grids in E3SM selectively inserts pentagons where needed. Only the first two approaches (chevrons and natural) work with unstructured variable-resolution grids. The pentagon approach only works with cubed-sphere grids.

Meta data files: (not used directly by the E3SM model, but used by the various pre- and post-processing utilities

Exodus: <grid>.g
- NetCDF file with grid information in the Exodus format: https://github.com/gsjaardema/seacas#exodus
- This file contains a description of the spectral elements, but not the GLL or PG cells within each spectral element.
- Can be used by TempestRemap to make mapping files for GLL grids for a given np.
- Can be used by TempestRemap to make SCRIP files for PG grids.
"latlon": <grid>_latlon_<date>.nc
- This file contains the GLL node locations and the native grid and the subcell grid connectivity
- Used by CAM's interpic_new to generate initial condition files
- Used by Paraview and Visit to plot native grid CAM-SE output
SCRIP: <grid>_scrip_<date.nc>
- This file contains the GLL node locations, the dual grid cell vertices and the dual grid connectivity
- Used by ESMF to create mapping files.

Tools:

TempestRemap: https://github.com/ClimateGlobalChange/tempestremap.
1. Used to make Exodus files (GLL and physgrid) and to make SCRIP files (physgrid only).
2. Used to make mapping files for GLL grids directly from the Exodus file.
Obsolete: For quasi-uniform cubed-sphere grids, E3SM currently uses the "pentagons" option to construct the dual grid for making ESMF mapping files. To create these meta data grid files, we use a utility distributed with the E3SM code - see e3sm/components/homme/test/tool.
Obsolete: For RRM grids, E3SM v1 used the "cheveron" option to construct the dual grid for making ESMF mapping files. To create these meta data grid files we use a Matlab program that performs a Newton interation. This matlab code is in the E3SM PreAndPostProcessingScripts repo under regridding/spectral_elements_grid_utilities/

Using TempestRemap to create Exodus and SCRIP files:

To create the Exodus file. For cubed sphere grids with a resolution of "NE":
1. GenerateCSMesh --alt --res $NE --file ne${NE}.g
2. For RRM grids, the Exodus .g file is part of the model configuration and is stored along with the initial condition files in the E3SM input data server, inputdata/atm/cam/inic/homme
To create a SCRIP file for PG2 grids from the Exodus file:
1. GenerateVolumetricMesh --in ne${NE}.g --out ne${NE}pg2.g --np 2 --uniform
2. ConvertMeshToSCRIP --in ne${NE}pg2.g --out ne${NE}pg2.scrip.nc
For E3SM V2, the physics is running on the "PG2" grid. This means most output, and all coupled model mapping files need to work with PG2 grids.
1. Mapping files should be created using the SCRIP file