CAM-SE works with data stored at Gauss-Lobatto-Legendre (GLL) nodes. These nodes can be interpreted in several ways shown in the three grids below. In all three grids, the GLL nodes are the same (the green dots in the figure). Internally, the dycore uses the CAM-SE native grid connectivity, but the rest of CAM and ACME E3SM has no knowledge of the grid connectivity and only sees data at GLL nodes and their associated area weight. The other other types of connectivity are not used within the model, but are sometimes used by pre- and post-processing tools.
...
Give any SE grid with N elements, the number of unique GLL nodes will be N*(np-1)^2 + 2
To map data between SE native grids and FV grids, TempestRemap should be used (see Transition to TempestRemap for Atmosphere grids ). SE GLL data is not naturally represented as FV type cell averages, and thus mapping tools which only work with cell centered data (ESMF, SCRIP) should be avoided.
The PG2 "physgrid"
In E3SM v2, we have the option to run the physics on a finite volume grid, where each spectral element cell is further divide into subcells. PG2 computes the physics on a 2x2 FV grid within each spectral element (shown below). On the physgrid, the physics columns are located at the cell center (green dots below), but should be considered cell average values over each FV cell, rather than nodal values at the green points.
...
Give a SE grid with N elements, the number of unique physgrid nodes will be N*(pg)^2
The physgrid is a FV grid and can be used with any tool that works with cell centered FV data (such as ESMF map file algorithms).
SE "subcell" grid
Most analysis tools and other tools cannot handle higher order elements such as used in CAM-SE. For these codes, we produce metadata that divide each spectral element into subcells. This subcell grid is shown below. The GLL nodes are the vertices of the subcell grid. This meta data can be used by Paraview and Visit to plot native grid CAM-SE output. The metadata is stored in a grid template file that typically has a file name of the form "<gridname>_latlon_<date>.nc" in the grids directory of the ACME E3SM inputdata server. From Euler's polyhedron formula, we know if there are N physics columns (GLL nodes), there will be N-2 subcells (all quads) and N/2-1 subcell edges.
SE "dual" grid
To remap data from the SE grid to other grids, ACME relies on ESMF mapping utilities. ESMF's conservative remapping algorithm can currently only map cell centered data. For every point in the mesh, ESMF treats that point as a cell average value, and requires that we specify its cell boundaries as a polygon. These polygons are the dual grid to the SE subcell grid. The metadata is stored in a grid template file in "SCRIP" format, that typically has a file name of the form "<gridname>_scrip-<date>.nc" in the grids directory of the ACME inputdata server.
This dual grid is needed as an input to the ESMF mapping utility to construct conservative mapping files. It is not needed nor used by the ACME model or our post processing tools. Constructing Some applications require treating the GLL nodal data as FV cell centered data. To do this, we need to construct the "dual grid" of the GLL subcell grid, resulting in a grid where each polygon contains a single GLL nodal point. This is not recommended and as one can see below is an unnatural representation of the GLL nodal data.
Constructing a SE dual grid is non-trivial and we have explored several options. :
For conservative remappingconservation, the areas of these polygons must match the weight of each GLL node. If we construct this dual grid in the usual way, by connecting the centers of all the subcells shown in the subcell grid, the areas of the cells will in general not match the GLL weights. We thus have to perform an iteration, similar to spring dynamics, tweaking the polygons until the areas are correct. We have several different algorithms to do this (examples shown below). The resulting polygons can be a little odd. First plot below (chevrons): we allow pentagons and hexagons so that the algorithm will converge faster and to more uniform cells, but this means that some of them will be slightly non-convex as some of the hexagons turn into chevrons. ESMF can handle non-convex cells, but other utilities may require they be convex. Second plot below (natural): If we only dual grid vertices at cell centers of the subcell grid, we get convex polygons, but they are less regular. Third plot below (pentagons): The dual grid files currently used for cubed-sphere grids in ACME E3SM selectively inserts pentagons where needed. Only the first two approaches (chevrons and natural) work with unstructured variable-resolution grids. The pentagon approach only works with cubed-sphere grids.
Meta data files: (not used directly by the
...
E3SM model, but used by the various pre- and post-processing utilities
- <grid>_latlon_<date>.nc
- This file contains the GLL node locations and the native grid and the subcell grid connectivity
- Used by CAM's interpic_new to generate initial condition files
- Used by Paraview and Visit to plot native grid CAM-SE output
- Used by TempestRemap to create mapping files (when specifying the Jacobian to override Tempests internal reference element map)
- <grid>_scrip_<date.nc>
- This file contains the GLL node locations, the dual grid cell vertices and the dual grid connectivity
- Used by ESMF to create mapping files.
...
- For quasi-uniform cubed-sphere grids, ACME E3SM currently uses the "pentagons" option to construct the dual grid for making ESMF mapping files. To create these meta data grid files, we use a utility distributed with the ACME E3SM code - see acmee3sm/components/homme/test/templatestool.
- For RRM grids, ACME currently uses E3SM v1 used the "cheveron" option to construct the dual grid for making ESMF mapping files. To create these meta data grid files we use a Matlab program that performs a Newton interation. This matlab code is in the ACME E3SM PreAndPostProcessingScripts repo under regridding/spectral_elements_grid_utilities/
Future Plans
We note that the TempestRemap package uses an algorithm that understands both SE and FV grids (Ullrich & Taylor, MWR 2015) and can produce map files directly from the CAM-SE Native grid. It only needs the SE Native grid (the first grid shown on this page) and does not need any type of dual grid. These mapping files are more accurate since they take advantage of the SE discretization. If ACME atmosphere adopted TempestRemap mapping files, we will not need to use the SE dual grid.
...
See also CAM-FV Grid Overview
...