W16 Physics Grid Design Document
The Design Document page provides a description of the algorithms, implementation and planned testing including unit, verification, validation and performance testing. Please read Step 1.3 Performance Expectations that explains feature documentation requirements from the performance group point of view.
Design Document
In the table below 4.Equ means Equations and Algorithms, 5.Ver means Verification, 6.Perf - Performance, 7. Val - Validation, 
- competed, 
- in progress, 
- not done
Overview table for the owner and an approver of this feature
1.Description | This is a descriptive long title |
| 2.Owner | Andrew Bradley |
| 3.Created | 14 Aug 2019 |
| 4.Equ | |
| 5.Ver | |
| 6.Perf | |
| 7.Val | |
| 8.Approver | |
| 9.Approved Date | |
| V2.0 |
Title: Physics Grid (Physgrid) With High-Order, Property-Preserving Remap
Requirements and Design
Date: July-August 2019
Summary
The high-level problem is to run the physics parameterizations on the same element grid as dynamics but a different nodal grid. For brevity, this capability is called physgrid. This feature requires modifying infrastructure to support two different grids, and write software to implement high-order, property-preserving remap algorithms.
There are two primary motivations for physgrid. First, on a quasi-uniform cubed sphere mesh, physics columns are given roughly uniform areas. Second, for nphys < 3, we can expect speedup of the physics parameterizations top-level calls of up to 9/nphys^2.
A narrative of development with extended results is available here: /wiki/spaces/COM/pages/1010171942
Requirements
Requirements for infrastructure include the following:
1. Generate domain and topography files for physgrid and put these on the input server.
2. Update configuration XML files to be able to express physgrid parameters and list default files.
Requirements for remap include the following:
1. In each element (and thus also globally), remap should conserve mass prior to the DSS. After the DSS, mass should be conserved globally.
2. In each element, remap of a tracer should not introduce new extrema prior to the DSS. After the DSS, there should be no new global extrema.
3. In each element, remap of a tracer should be tracer consistent. The DSS maintains tracer consistency.
4. Remap of horizontal velocity should avoid problems at the poles.
5. Algorithms should work seamlessly on quasi-uniform and RRM SE quad meshes. In particular, we strongly prefer to rely on element-neighbor data only through HOMME's standard halo exchange patterns.
6. For nphys > 2, order of accuracy (OOA) in the l2 norm should be at least 2 on a smooth test function.
Algorithmic Formulations
The following design document, which we shall prefer to as the "latex design doc," describes algorithms: Physgrid High-Order Remap
Design and Implementation
The code is organized as follows:
- homme/src/share/gllfvremap_mod.F90: Implementation of
algorithms. The gllfvremap_mod module exports the following primary
public API:- gfr_init: Initialize data structures and geometry data. Called in dyn_comp.F90.
- gfr_finish: Deallocate memory. Called in stepon.F90.
- gfr_fv_phys_to_dyn_topo: Record FV topography data and remap it to GLL. Later, FV topography data can be either remapped from GLL to FV, which provides a consistent average area within an FV cell independent of basis, or provided as the raw data read from the topography file. Called in inidat.F90.
- gfr_dyn_to_fv_phys: Remap T, u, v, omega_p, q, p_s, phi_s from GLL to FV bases. Called in dp_coupling.F90.
- gfr_fv_phys_to_dyn: Remap T, u, v, q tendencies or states from FV to GLL bases. Called in dp_coupling.F90.
- To support testing outside of the module, it provides a few additional public functions. Inside the module, the public function gfr_test iterates through a number of unit tests (test type 1) that test element-level area and mass conservation, shape preservation, reversibility of maps when expected, and order of accuracy (OOA); we summarize these quantities as properties and OOA.
- homme/src/share/gllfvremap_test_mod.F90: Tests (type 2) of global properties and OOA using the public API.
- homme/src/test_src/dcmip16_wrapper.F90: DCMIP 2016 moist baroclininc instability test (type 3) using physgrid. Implemented in dcmip2016_test1_pg and dcmip2016_test1_pg_forcing. Tests the public API.
This software is operational and fully testable in standalone HOMME. It is operational and type 1 tests can be called in principle in the E3SM build.
Type 2 tests can be run in a loop over n_e to produce convergence plots. The type-3 test permits us to study instantaneous snapshots of, e.g., q_v in a nontrivial setting but for which only a few compute nodes and approximately 10 minutes of computation are needed.
Planned Verification and Unit Testing
The previous section describes the types of tests. Results of the order of accuracy test are in the latex design doc, Figure 2.
Planned Validation Testing
We are running 6 years of the ne30 FC5AV1C-L compset with the default ne30np4 and new ne30pg2 grid configurations.
Planned Performance Testing
We will record and analyze timers in our F compset runs on Cori-KNL.