W10 NH Dycore with SL Transport Design Document

Owned by
Renata McCoy
Last updated: Apr 02, 2020
5 min read

The Design Document page provides a description of the algorithms, implementation and planned testing including unit, verification, validation and performance testing. Please read  Step 1.3 Performance Expectations that explains feature documentation requirements from the performance group point of view. 

Design Document

 Click here for instructions to fill up the table below ......

The first table in Design Document gives overview of this document, from this info the Design Documents Overview page is automatically created.

In the overview table below 4.Equ means Equations and Algorithms, 5.Ver means Verification, 6.Perf - Performance, 7. Val - Validation

  • Equations: Document the equations that are being solved and describe algorithms
  • Verification Plans: Define tests that will be run to show that implementation is correct and robust. Involve unit tests to cover range of inputs as well as benchmarks.
  • Performance expectations: Explain the expected performance impact from this development
  • Validation Plans: Document what process-based, stand-alone component, and coupled model runs will be performed, and with what metrics will be used to assess validity

Use the symbols below (copy and paste) to indicate if the section is in progress or done or not started.

In the table below 4.Equ means Equations and Algorithms, 5.Ver means Verification, 6.Perf - Performance, 7. Val - Validation,   (tick) - competed, (warning) - in progress, (error) - not done


Overview table for the owner and an approver of this feature

1.Description

NH Dycore with SL Transport

2.OwnerMark Taylor
3.Created 
4.Equ(tick)
5.Ver(tick)
6.Perf(tick)
7.Val(tick)
8.Approver
9.Approved Date
V2.0Pending
 Click here for Table of Contents ...

Table of Contents




Title: NH Dycore with SL Transport

Requirements and Design

E3SM Water Cycle Group

Date:  

Summary

E3SM v1 uses a hydrostatic dynamical core ("preqx") with Eulerian transport from the HOMME package.  For E3SM v2, we propose to adopt HOMME's "theta-l" nonhydrostatic dynamical core with Semi-Lagrangian transport.   There are three motivations for this work:  (1) Allow E3SM to run at resolutions higher then 10km  where the nonhydrostatic dynamics start to become significant.  (2)  Improve the accuracy of both the dynamics and tracer transport algorithms  and (3) improve the computational performance of the full dycore in configurations relevant to the E3SM v2 science campaigns.   

Contributors: Peter Bosler, Andrew Bradley, Oksana Guba , Mark Taylor, Andrew Steyer (Unlicensed), Paul Ullrich


Requirements

  • Solve the nonhydrostatic equations with a HEVI approach
  • Support a mass coordinate hydrostatic option to ease integration into E3SM
  • Improve accuracy over E3SM v1 dycore  (  E3SM v1 dycore = "preqx" + Eulerian transport)
  • Improve performance over E3SMv1 dycore


Algorithmic Formulations

The formulation of the nonhydrostatic equations is documented in a journal paper to be submitted to JAMES:  https://www.overleaf.com/read/bqycjpqjthnc.   Details include:

  • Nonhydrostatic equations in spherical geometry with a Lorenz staggered vertical discretization.
  • Conserves a discrete energy when coupled with HOMME's existing mimetic spectral element horizontal discretization.
  • Primitive variable formulation
  • Supports terrain following mass or height coordinates and for both Eulerian or vertically Lagrangian discretizations.
  • Energy conservation is obtained via a term-by-term balance in the kinetic, internal and potential energy budgets, ensuring an energy-consistent discretization with no spurious sources of energy.

The SL transport algorithm is documented in two recently accepted SISC papers:

  • The property preservation methods are described and analyzed in this article (open access) and are implemented and unit tested in the CEDR subpackage of COMPOSE.
  • For the advection step, a cell-integrated incremental remap SL method is described in a paper currently in the production stage; that will be linked when it's available. The geometry calculations are implemented and unit tested in the SIQK subpackage of COMPOSE.

Further details are as follows:

  • SL methods minimize the number of communication rounds and volume for tracer transport per unit of simulated time, relative to other methods.
  • Tracer transport must be shape preserving, mass conserving, and mass tracer consistent. CEDR provides these properties.
  • For the advection step, we have opted to use an interpolation method rather than a cell-integrated method. This will be described in a paper we are currently drafting. Since a reference is not available yet, we will describe the key ideas here:
    1. CEDR handles all aspects of property preservation efficiently. It is needed for CISL as well as ISL.
    2. Interpolation SL (ISL) methods are the most efficient SL methods for four reasons:
      1. The remap formula is computationally parsimonious.
      2. An optimal implementation of communication is parsimonious. ISL requires ~4x data movement than CISL.
      3. It is easy to go to high order because edges do not have to be represented at high order, since there are no explicit edges.
      4. ISL accuracy can easily be boosted with p-refinement.
    3. Thus, ISL is a better choice than CISL: it is much faster, and it can also provide extremely high accuracy if desired.
    4. The key challenge is to find interpolants yielding a stabilized ISL on an SE mesh. We have done this, and these will be described in the paper. Roughly, interpolation formulas supported by a single spectral element are searched for the highest-order one that is unconditionally stable on the periodic transport problem.


Design and Implementation


The code has been implemented in the HOMME dynamical core as a non-default compile time option:   https://github.com/E3SM-Project/E3SM/tree/master/components/homme

The NH dycore is build by specifying the "theta-l" target, as opposed to the default "preqx" target.   The "theta-l" model reuses much of the existing E3SM code, with all new code specific to "theta-l" placed in the  https://github.com/E3SM-Project/E3SM/tree/master/components/homme/src/theta-l directory.   The implementation follows the "preqx" model, using the same infrastructure, but providing replacements for key modules:  prim_advance_exp() is the high level subroutine which is the high level driver for the dynamics step ( Runge-Kutta IMEX for the adiabatic component of the flow followed by forward euler hyperviscosity operator).   The Runge-Kutta stages are implemented in the compute_andor_apply_rhs() subroutine.   Other new routines include advance_hypervis() and vert_remap() which are identical to the versions from "preqx", except for being updated to take into account the different prognostic variables used by "theta-l".   The interface to the thermodynamic variables is provided by element_ops.F90.  

SL is implemented in src/share/compose as a library. It is built independent of compile-time sizes such as np and qsize, so the library is built just once regardless of number of HOMME executable variants built. Currently, code is imported to src/share/compose/compose_cedr.cpp and src/share/compose/compose_slmm.cpp from the COMPOSE repo, but we intend to use a submodule eventually. The SL transport algorithm is turned on via namelist options.  COMPSETS will be created that use the recommended options with the PR submitted with this design document.


Planned Verification and Unit Testing 


Plans for verification and unit testing was developed and approved under DOE Scidac transport and nonhydrostatic projects.   They include unit tests for all new aspects of the algorithms, verification using DCMIP test cases and the addition of functional tests to the E3SM test suite.   The results are described W10 NH Dycore with SL Transport Verification Phase 1


Planned Validation Testing 


Validation will be performed with E3SM v1 F compsets, using the standard E3SM diagnostics package.  The results are described W10 NH Dycore with SL transport Validation Phase 1


Planned Performance Testing 

As this work is limited to the HOMME dynamical core, we will use the E3SM Performance Group's /wiki/spaces/EPG/pages/905969931.   The results are described W10 NH Dycore with SL transport Performance Phase 1