Why and how to migrate to updated grids and mapfiles

This page is under construction...explanation and links will be added in the coming days and weeks...feedback welcome!

ACME model output on unstructured grids is often remapped in a post-processing step to structured lat/lon analysis grids for use by most of our analysis tools.  There are numerous problems with these analysis grids employed for remapping by ACME (and CESM) prior to 20150901.  These small flaws or limitations propagate into the weights produced by the weight-generation utility. The flawed weights produce undesirable outcomes (loss of precision, gaps) when converting from source to destination maps. All tested regridders correctly apply the weights they are supplied, and migrating to improved grids (and to mapfiles generated from those grids, e.g., by ESMF_RegridWeightGen or TempestRemap) automatically improves both the numerical accuracy and the data and metadata completeness and consistency of the files produced by the regridding procedure. None of the problems described below affect the accuracy of the model results on the native grid. The affected grids include many FV (plain and staggered) and Gaussian grids known to be used for ACME analysis, mapfiles produced from those grids, and all mapfiles employing bilinear interpolation. The new grids improve the accuracy of diagnostics and the aesthetics of plots produced from regridded files.

Known Issues:  We identified five issues (four numerical flaws and one empty field) in the pre-migration (pre-20150901) gridfiles and mapfiles.

1. ACME adopted flawed FV-style gridfiles that omitted a small strip of longitude to the east of Greenwich. This problem was identified independently by Charles Doutriaux and myself. For FV 129x256, this amounted to 0.2% of global area, that might appear as a gap or blank strip when plotted. Maps based on the flawed grids somehow reapportion area so that total area is conserved (4*pi sr), yet this necessarily redistributes weights from their true positions. This causes a mismatch between "area"- and "gw"- weighted statistics. The solution is to generate grids that center Greenwich in the first zonal gridcell, and to base maps on those grids. With these grids, the gap disappears and "area"- and "gw"-weighted statistics agree to double-precision. 

2. SCRIP introduced, and CESM and ACME inherited, coordinate storage in double precision (yay!). Unfortunately, every Gaussian grid that I have examined has grid center latitudes (= sine of the Gaussian quadrature points) accurate to no greater than eight digits. This problem appears in files in the SCRIP distribution, and in all files produced by NCL that I have examined. The solution is to base latitudes on quadrature points (i.e., Legendre solutions) computed to full double precision. NCO now generates SCRIP-format Gaussian grids accurate to sixteen digits, the best that double precision can reach.

3. All SCRIP and CESM-maintained Gaussian grids that I have examined (T42, T62, and T85) infer gridcell interfaces as midpoints between Gaussian quadrature points/angles. Software then infers the gridcell areas from gridcell interfaces. The problem is that interfaces defined by the midpoint rule will always produce areas inconsistent with those implied by the quadrature weights. This causes a mismatch between "area"- and "gw"- weighted statistics. NCO now uses Newton-Raphson iteration (instead of the quadrature midpoints) to determine the gridcell interface location that exactly matches areas determined by the (now double-precision) Gaussian weights. The Newton-Raphson iteration moves interfaces by, typically, a few tenths of a degree (for moderate resolution Gaussian grids) from their previous locations as quadrature midpoints. With these grids, "area"- and "gw"-weighted statistics are consistent and agree to double-precision.

4. Some equi-angular lat/lon grids historically produced and used in the CESM community utilize the continuous form of the weighting function (i.e., cosine(lat)) evaluated on the discretized grid, rather than the exact discretized weight function (i.e., the difference of the sine of the latitudes bounding the zone). And some of these equi-angular lat/lon grids are used as staggered/offset grids for FV dynamics variables (e.g., U, V). All equi-angular lat/lon grids must have the same functional form for weights as the FV grid itself, since they are simply offsets of eachother. Without this correction, statistics of variables computed on the FV dynamics grid may be misdiagnosed (fxm: last sentence not yet verified). Users of all equi-angular grids, including staggered FV grids, should ensure their grids were produced with the correct weighting function. One way to do this is to verify the "area" variable, if present, exactly matches the area between the gridcell interfaces. If unsure, consider generating a new grid with the instructions below, or ask me and I will be happy to generate one for you and keep it in a standard location where others may access it.

5. For historical reasons, mapfiles generated by ESMF_RegridWeightGen (up to and including version 6.3.0rp1) using bilinear interpolation do not include complete output grid information. In particular, they lack gridcell area, because the ESMF team thought area would not be desired by users employing non-conservative mapping. However, it is helpful to include area so long as users understand that interpolative maps are non-conservative. The new mapfiles include gridcell area for bilinear interpolation maps. 

6. Great circle arcs vs. lat/lon arcs, latitude-weights, and --user_areas...fxm.

Why Migrate?

The new grids and mapfiles address these problems, which have always existed in ACME and its predecessors (CESM, CCSM, CCM), and therefore cannot be too severe. The numerical flaws explained above can be thought of as fuzziness at the level of a few tenths of a degree in geo-referencing regridded data to the native model grid. These location errors produce only small (<< 1%) errors in regional or global statistics. So, why migrate? One aim is that diagnostics and observational evaluations with regridded data (often much more intuitive to visually evaluate than native SE grids) produce the same answers (to double precision whenever possible) as statistics computed on the native model grid. Without migration, agreement between native and regridded statistics beyond single precision is a matter of luck and coincidence, not determinism and reproducibility. As ACME grids shift to ~1/4 degree and finer, it becomes even more important to exploit the full double precision accuracy that software can guarantee when supplied with accurate grids.

We generated new FV-type analysis grids with NCO (version 4.5.2 or later) and commands shown in the tables below. These grids have replaced the flawed grids on the ACME inputdata server: https://acme-svn2.ornl.gov/acme-repo/acme/mapping/grids. Gaussian grids were never distributed within ACME and so the improved versions are not currently stored on ACME's inputdata server. However, Gaussian grids have been used extensively by the CESM community. Their replacements are available on rhea (/lustre/atlas/proj-shared/cli115/zender/grids) and yellowstone (/glade/p/work/zender/grids).

Note that ACME gridfiles and mapfiles are date-stamped. The new grid and mapfiles were produced on or after 20150724. Cubed-sphere gridfiles produced before 20150724 are still considered good. Many other atmosphere gridfiles produced before 20150724 contain the flaws mentioned above. Gridfile flaws propagate to mapfiles so in most cases mapfiles must be updated too

The netCDF Operator ncks generates SCRIP-format gridfiles for select grid types, including equi-angular, FV, and Gaussian rectangular latitude/longitude grids. Full documentation is at http://nco.sf.net/nco.html#grid. For convenience, we sumarrize that functionality here. Options pertinent to the grid geometry and metadata are passed to NCO via key-value pairs prefixed by "--rgr". Pass at least six key-value pair arguments to create a fully explicitly specified, well-annotated grid. The six arguments, and their corresponding keys, are: grid title (grd_ttl), grid filename (grid), number of latitudes (lat_nbr), number of longitudes (lon_nbr), latitude grid type (lat_typ), and longitude grid type (lon_typ). Four other arguments (the NSEW bounding box) are necessary to construct regular regional (non-global) grids, but for now we focus on global grids.

The lat_typ options for global grids are "eqa" for Equi-angular, "fv" for Finite Volume, and "gss" for Gaussian. An FV grid is nearly identical to an Equi-angular grid, with some important data and metadata differences at the poles. FV grids must have an odd number of latitudes because the first and last latitudes have special properties (and span half the width
of the other latitudes), whereas the Equiangular grid may have any number of latitudes. Gaussian grids must have an even number of latitudes. The lon_typ options for global grids are "grn_ctr" and "180_ctr" for the first gridcell centered at Greenwich or 180 degrees, respecitvely. And "grn_wst" and "180_wst" for Greenwich or 180 degress lying on the western edge of the first gridcell.

All grids created for this migration thus far are filename date-stamped with 20150901 to facilitate recognizing updated gridfiles and mapfiles. Full creation metadata is in the file header. Any valid netCDF file may be named as the source (e.g., in.nc). It will not be altered. The destination file (foo.nc) will be overwritten. Its contents are immaterial. Add a "-D 7" option to increase the output verbosity and check, e.g., the precision of normalization of area and latitude weights. The API for creating grids is primitive (e.g., having to repeat "--rgr") because it was quickly bolted-on to NCO. We may improve and extend the NCO API to specify other grids and maps in the future. Important updates will be noted here.

Example: 

 We analyzed the global mean net TOA solar radiation for January, 1979 from an ne30 simulation on the native grid and then with old and mapfiles. Tests were performed with commands of the form "ncwa -w area -v FSNT in.nc out.nc" and "ncwa -w gw -v FSNT in.nc out.nc" where in.nc = famipc5_ne30_v0.3_00003.cam.h0.1979-01.nc for the native grid and a regridded version of that for the other mapfiles. The native grid analysis is independent of mapfile and is show for clarity. Analysis grids produced with conservative regridding should exactly reproduce the native grid results. The "area" and "gw" rows show this is only true for the new map with the "–user_area" switch enabled.