W16 Physics Grid Design Document

Owned by Renata McCoy
Last updated: Apr 27, 2020
4 min read

The Design Document page provides a description of the algorithms, implementation and planned testing including unit, verification, validation and performance testing. Please read  Step 1.3 Performance Expectations that explains feature documentation requirements from the performance group point of view. 

Design Document

 Click here for instructions to fill up the table below ......

The first table in Design Document gives overview of this document, from this info the Design Documents Overview page is automatically created.

In the overview table below 4.Equ means Equations and Algorithms, 5.Ver means Verification, 6.Perf - Performance, 7. Val - Validation

  • Equations: Document the equations that are being solved and describe algorithms
  • Verification Plans: Define tests that will be run to show that implementation is correct and robust. Involve unit tests to cover range of inputs as well as benchmarks.
  • Performance expectations: Explain the expected performance impact from this development
  • Validation Plans: Document what process-based, stand-alone component, and coupled model runs will be performed, and with what metrics will be used to assess validity

Use the symbols below (copy and paste) to indicate if the section is in progress or done or not started.

In the table below 4.Equ means Equations and Algorithms, 5.Ver means Verification, 6.Perf - Performance, 7. Val - Validation,   (tick) - competed, (warning) - in progress, (error) - not done


Overview table for the owner and an approver of this feature

1.Description

This is a descriptive long title
2.OwnerAndrew Bradley
3.Created14 Aug 2019
4.Equ(tick)
5.Ver(tick)
6.Perf(tick)
7.Val(tick)
8.Approver
9.Approved Date
V2.0
 Click here for Table of Contents ...

Table of Contents




Title: Physics Grid (Physgrid) With High-Order, Property-Preserving Remap

Requirements and Design

Date: July-August 2019 

Summary

The high-level problem is to run the physics parameterizations on the same element grid as dynamics but a different nodal grid. For brevity, this capability is called physgrid. This feature requires modifying infrastructure to support two different grids, and write software to implement high-order, property-preserving remap algorithms.

There are two primary motivations for physgrid. First, on a quasi-uniform cubed sphere mesh, physics columns are given roughly uniform areas. Second, for nphys < 3, we can expect speedup of the physics parameterizations top-level calls of up to 9/nphys^2.

A narrative of development with extended results is available here: /wiki/spaces/COM/pages/1010171942

Requirements

Requirements for infrastructure include the following:

1. Generate domain and topography files for physgrid and put these on the input server.

2. Update configuration XML files to be able to express physgrid parameters and list default files.

Requirements for remap include the following:

1. In each element (and thus also globally), remap should conserve mass prior to the DSS. After the DSS, mass should be conserved globally.

2. In each element, remap of a tracer should not introduce new extrema prior to the DSS. After the DSS, there should be no new global extrema.

3. In each element, remap of a tracer should be tracer consistent. The DSS maintains tracer consistency.

4. Remap of horizontal velocity should avoid problems at the poles.

5. Algorithms should work seamlessly on quasi-uniform and RRM SE quad meshes. In particular, we strongly prefer to rely on element-neighbor data only through HOMME's standard halo exchange patterns.

6. For nphys > 2, order of accuracy (OOA) in the l2 norm should be at least 2 on a smooth test function.

Algorithmic Formulations

The following design document, which we shall prefer to as the "latex design doc," describes algorithms: Physgrid High-Order Remap

Design and Implementation

The code is organized as follows:

  • homme/src/share/gllfvremap_mod.F90: Implementation of
    algorithms. The gllfvremap_mod module exports the following primary
    public API:
    • gfr_init: Initialize data structures and geometry data. Called in dyn_comp.F90.
    • gfr_finish: Deallocate memory. Called in stepon.F90.
    • gfr_fv_phys_to_dyn_topo: Record FV topography data and remap it to GLL. Later, FV topography data can be either remapped from GLL to FV, which provides a consistent average area within an FV cell independent of basis, or provided as the raw data read from the topography file. Called in inidat.F90.
    • gfr_dyn_to_fv_phys: Remap T, u, v, omega_p, q, p_s, phi_s from GLL to FV bases. Called in dp_coupling.F90.
    • gfr_fv_phys_to_dyn: Remap T, u, v, q tendencies or states from FV to GLL bases. Called in dp_coupling.F90.
    • To support testing outside of the module, it provides a few additional public functions. Inside the module, the public function gfr_test iterates through a number of unit tests (test type 1) that test element-level area and mass conservation, shape preservation, reversibility of maps when expected, and order of accuracy (OOA); we summarize these quantities as properties and OOA.
  • homme/src/share/gllfvremap_test_mod.F90: Tests (type 2) of global properties and OOA using the public API.
  • homme/src/test_src/dcmip16_wrapper.F90: DCMIP 2016 moist baroclininc instability test (type 3) using physgrid. Implemented in dcmip2016_test1_pg and dcmip2016_test1_pg_forcing. Tests the public API.

This software is operational and fully testable in standalone HOMME. It is operational and type 1 tests can be called in principle in the E3SM build.

Type 2 tests can be run in a loop over n_e to produce convergence plots. The type-3 test permits us to study instantaneous snapshots of, e.g., q_v in a nontrivial setting but for which only a few compute nodes and approximately 10 minutes of computation are needed.

Planned Verification and Unit Testing 

The previous section describes the types of tests. Results of the order of accuracy test are in the latex design doc, Figure 2.

Planned Validation Testing 

We are running 6 years of the ne30 FC5AV1C-L compset with the default ne30np4 and new ne30pg2 grid configurations.

Planned Performance Testing 

We will record and analyze timers in our F compset runs on Cori-KNL.