CAM-SE works with data stored at Gauss-Lobatto-Legendre (GLL) nodes. These nodes can be interpreted in several ways shown in the three grids below. In all three grids, the GLL nodes are the same (the green dots in the figure). Internally, the dycore uses the CAM-SE native grid connectivity, but the rest of CAM and ACME has no knowledge of the grid connectivity and only sees data at GLL nodes and their associated area weight. The other other types of connectivity are not used within the model, but are sometimes used by pre- and post-processing tools.
...
Here is a low-resolution CAM-SE grid. This is a "ne4np4" grid, meaning that it has 4x4 spectral elements in each cube face (the "ne" value), and each spectral element has a grid of 4x4 (the "np" value) Gauss-Lobatto points inside it. The blue lines show the edges of the spectral elements. CAM-SE is collocated, meaning that all variables (U,V,T, etc...) are carried around on the GLL nodes. CAM physics is computed in columns at GLL nodes. Each column has a Gauss-Lobatto weight associated with it, which can also be interpreted as an area associated with each node. The sum of these weights is 4pi (the area of the unit sphere). One can see from the figure that GLL nodes are not quite equally spaced - they cluster at the edges of the spectral elements.
Give any SE grid with N elements, the number of unique GLL nodes will be N*(np-1)^2 + 2
The PG2 "physgrid"
In E3SM v2, we have the option to run the physics on a finite volume grid, where each spectral element cell is further divide into subcells. PG2 computes the physics on a 2x2 FV grid within each spectral element (shown below). On the physgrid, the physics columns are located at the cell center (green dots below), but should be considered cell average values over each FV cell, rather than nodal values at the green points.
For v2 simulations, most output will be on the PG2 physgrid.
Give a SE grid with N elements, the number of unique physgrid nodes will be N*(pg)^2
SE "subcell" grid
Most analysis tools and other tools cannot handle higher order elements such as used in CAM-SE. For these codes, we produce metadata that divide each spectral element into subcells. This subcell grid is shown below. The GLL nodes are the vertices of the subcell grid. This meta data can be used by Paraview and Visit to plot native grid CAM-SE output. The metadata is stored in a grid template file that typically has a file name of the form "<gridname>_latlon_<date>.nc" in the grids directory of the ACME inputdata server. From Euler's polyhedron formula, we know if there are N physics columns (GLL nodes), there will be N-2 subcells (all quads) and N/2-1 subcell edges.
SE "dual" grid
To remap data from the SE grid to other grids, ACME relies on ESMF mapping utilities. ESMF's conservative remapping algorithm can currently only map cell centered data. For every point in the mesh, ESMF treats that point as a cell average value, and requires that we specify its cell boundaries as a polygon. These polygons are the dual grid to the SE subcell grid. The metadata is stored in a grid template file in "SCRIP" format, that typically has a file name of the form "<gridname>_scrip-<date>.nc" in the grids directory of the ACME inputdata server.
...
For conservative remapping, the areas of these polygons must match the weight of each GLL node. If we construct this dual grid in the usual way, by connecting the centers of all the subcells shown in the subcell grid, the areas of the cells will in general not match the GLL weights. We thus have to perform an iteration, similar to spring dynamics, tweaking the polygons until the areas are correct. We have several different algorithms to do this (examples shown below). The resulting polygons can be a little odd. First plot below (chevrons): we allow pentagons and hexagons so that the algorithm will converge faster and to more uniform cells, but this means that some of them will be slightly non-convex as some of the hexagons turn into chevrons. ESMF can handle non-convex cells, but other utilities may require they be convex. Second plot below (natural): If we only dual grid vertices at cell centers of the subcell grid, we get convex polygons, but they are less regular. Third plot below (pentagons): The dual grid files currently used for cubed-sphere grids in ACME selectively inserts pentagons where needed. Only the first two approaches (chevrons and natural) work with unstructured variable-resolution grids. The pentagon approach only works with cubed-sphere grids.
Meta data files: (not used directly by the ACME model, but used by the various pre- and post-processing utilities
...
- For quasi-uniform cubed-sphere grids, ACME currently uses the "pentagons" option to construct the dual grid for making ESMF mapping files. To create these meta data grid files, we use a utility distributed with the ACME code - see acme/components/homme/test/templates.
- For RRM grids, ACME currently uses the "cheveron" option to construct the dual grid for making ESMF mapping files. To create these meta data grid files we use a Matlab program that performs a Newton interation. This matlab code is in the ACME PreAndPostProcessingScripts repo under regridding/spectral_elements_grid_utilities/
Future Plans
We note that the TempestRemap package uses an algorithm that understands both SE and FV grids (Ullrich & Taylor, MWR 2015) and can produce map files directly from the CAM-SE Native grid. It only needs the SE Native grid (the first grid shown on this page) and does not need any type of dual grid. These mapping files are more accurate since they take advantage of the SE discretization. If ACME atmosphere adopted TempestRemap mapping files, we will not need to use the SE dual grid.
...