...
Date:
Summary
The purpose of this section is to summarize what capability is to be added to the ACME Ocean and Ice system through this design process. It should be clear what new code will do that the current code does not. Summarizing the primary challenges with respect to software design and implementation is also appropriate for this section. Finally, this statement should contain general statement with regard to what is “success.”E3SM v1 uses a hydrostatic dynamical core ("preqx") with Eulerian transport from the HOMME package. For E3SM v2, we propose to adopt HOMME's "theta-l" nonhydrostatic dynamical core with Semi-Lagrangian transport. There are three motivations for this work: (1) Allow E3SM to run at resolutions higher then 10km where the nonhydrostatic dynamics start to become significant. (2) Improve the accuracy of both the dynamics and tracer transport algorithms and (3) improve the computational performance of the full dycore in configurations relevant to the E3SM v2 science campaigns.
Contributors: Peter Bosler, Andrew Bradley, Oksana Guba , Mark Taylor, Andrew Steyer (Unlicensed), Paul Ullrich
Requirements
- Solve the nonhydrostatic equations with a HEVI approach
- Support a mass coordinate hydrostatic option to ease integration into E3SM
- Improve accuracy over E3SM v1 dycore ( E3SM v1 dycore = "preqx" + Eulerian transport)
- Improve performance over E3SMv1 dycore
Algorithmic Formulations
The formulation of the nonhydrostatic equations is documented in a journal paper to be submitted to JAMES: https://www.overleaf.com/read/bqycjpqjthnc. Details include:
- Nonhydrostatic equations in spherical geometry with a Lorenz staggered vertical discretization.
- Conserves a discrete energy when coupled with HOMME's existing mimetic spectral element horizontal discretization.
- Primitive variable formulation
- Supports terrain following mass or height coordinates and for both Eulerian or vertically Lagrangian discretizations.
- Energy conservation is obtained via a term-by-term balance in the kinetic, internal and potential energy budgets, ensuring an energy-consistent discretization with no spurious sources of energy.
The SL transport algorithm is documented in two recently accepted SISC papers:
- The property preservation methods are described and analyzed in this article (open access) and are implemented and unit tested in the CEDR subpackage of COMPOSE.
- For the advection step, a cell-integrated incremental remap SL method is described in a paper currently in the production stage; that will be linked when it's available. The geometry calculations are implemented and unit tested in the SIQK subpackage of COMPOSE.
Further details are as follows:
- SL methods minimize the number of communication rounds and volume for tracer transport per unit of simulated time, relative to other methods.
- Tracer transport must be shape preserving, mass conserving, and mass tracer consistent. CEDR provides these properties.
- For the advection step, we have opted to use an interpolation method rather than a cell-integrated method. This will be described in a paper we are currently drafting. Since a reference is not available yet, we will describe the key ideas here:
- CEDR handles all aspects of property preservation efficiently. It is needed for CISL as well as ISL.
- Interpolation SL (ISL) methods are the most efficient SL methods for four reasons:
- The remap formula is computationally parsimonious.
- An optimal implementation of communication is parsimonious. ISL requires ~4x data movement than CISL.
- It is easy to go to high order because edges do not have to be represented at high order, since there are no explicit edges.
- ISL accuracy can easily be boosted with p-refinement.
- Thus, ISL is a better choice than CISL: it is much faster, and it can also provide extremely high accuracy if desired.
- The key challenge is to find interpolants yielding a stabilized ISL on an SE mesh. We have done this, and these will be described in the paper. Roughly, interpolation formulas supported by a single spectral element are searched for the highest-order one that is unconditionally stable on the periodic transport problem.
Design and Implementation
The NH dycore is build by specifying the "theta-l" target, as opposed to the default "preqx" target. The "theta-l" model reuses much of the existing E3SM code, with all new code specific to "theta-l" placed in the https://github.com/E3SM-Project/E3SM/tree/master/components/homme/src/theta-l directory. The implementation follows the "preqx" model, using the same infrastructure, but providing replacements for key modules: prim_advance_exp() is the high level subroutine which is the high level driver for the dynamics step ( Runge-Kutta IMEX for the adiabatic component of the flow followed by forward euler hyperviscosity operator). The Runge-Kutta stages are implemented in the compute_andor_apply_rhs() subroutine. Other new routines include advance_hypervis() and vert_remap() which are identical to the versions from "preqx", except for being updated to take into account the different prognostic variables used by "theta-l". The interface to the thermodynamic variables is provided by element_ops.F90.
SL is implemented in src/share/compose as a library. It is built independent of compile-time sizes such as np and qsize, so the library is built just once regardless of number of HOMME executable variants built. Currently, code is imported to src/share/compose/compose_cedr.cpp and src/share/compose/compose_slmm.cpp from the COMPOSE repo, but we intend to use a submodule eventually. The SL transport algorithm is turned on via namelist options. COMPSETS will be created that use the recommended options with the PR submitted with this design document.
Planned Verification and Unit Testing
Unit tests will be developed to test all new aspects of the algorithms and formulation.
Verification will be done using DCMIP test cases
Functional tests will be added to the E3SM test suite
Planned Validation Testing
Validation will be performed with E3SM v1 F compsets, using the standard E3SM diagnostics package.
We run a 1 degree FC compset, FC5AV1C-L, simulation for 6 years and create 5-year (year 2 to 6) climatologies from it. We plot climatologies vs observations and vs default configuration. The default configuration consists of default hydrostatic dycore with default settings. Preliminary runs are described in /wiki/spaces/COM/pages/941096965 . There we produced runs with SL transport and theta-l dycore (hydrostatic only).
Planned Performance Testing
As this work is limited to the HOMME dynamical core, we will use the E3SM Performance Group's /wiki/spaces/EPG/pages/905969931. We expect the NH dycore with SL transport will be significantly faster at all processor counts and on all platforms.
...