W12: Improved Dust Aerosol Physics Validation Phase 1

This page should describe Validation Tests performed for this stand alone feature and should provide links to all the result pages.

+++ Information below is from another E3SM activity.  Replace with appropriate info for this effort +++

Summary


We run a 1 degree FC compset, FC5AV1C-L, simulation for 6 years and create 5-year (year 2 to 6) climatologies from it. We plot climatologies vs observations and vs default configuration. The default configuration consists of default hydrostatic dycore with default settings. Preliminary runs are described in /wiki/spaces/COM/pages/780664997 (with preqx dycore) and in /wiki/spaces/COM/pages/941096965 (with theta dycore), each relevant run also has entry for its climatologies.

SL transport was first evaluated in the preqx dycore using the E3SM v1 atmosphere at 1 degree. Results and some technical details are presented in /wiki/spaces/COM/pages/916324466 . The SL transport was shown to be 6x faster than Eulerian transport.

With theta dycore, we only explore the hydrostatic option for now. The preliminary configuration for theta-l dycore and SL transport is Run6 in /wiki/spaces/COM/pages/997752860 and is summarized below. We assess the climatology differences to be small overall, but below we note below specific differences that an expert should evaluate. In this validation test, the transport speedup is over 6x and the overall dycore speedup is approximately 3.7x for an FC compset.

Validation Test 1

Validation Test 1: preqx dycore and SL transport for 1 degree

Date last modified: Sept. 2018

Contributors: 

Provenance: commit 099c4dd, A. Bradley's fork of E3SM

Results: /wiki/spaces/COM/pages/780664997 , Run 19.


We ran FC5AV1C-L for 6 years with preqx dycore and SL transport and made 5-year climatologies. With diagnostics package, we compared climatologies of the new run to the default (preqx dycore, Eulerian transport, default E3SM settings) run from the same compset. Results are in https://portal.nersc.gov/project/acme/onguba/sl/sl-fastest.2369911-latlon-vs-f2/viewer/latitude-longitude/index.html for latlon plots and in https://portal.nersc.gov/project/acme/onguba/sl/sl-fastest.2369911-zonal-vs-f2/viewer/pressure-latitude/index.html for zonal plots. Our understanding from communicating with C. Golaz is that differences are relatively minor and are expected.

This task required new coupling mechanisms between homme and physics (ftypes). Tests for all coupling schemes are now in nightlies with baselines.


Validation Test 2

Validation Test 2: theta-l dycore and SL transport for 1 degree

Date last modified: June 2019

Contributors: 

Provenance: commit a0ff14648bf1f, A. Bradley's fork (PR #3017, now in master)

Results: /wiki/spaces/COM/pages/997752860 , Run6 only


We ran FC5AV1C-L for 6 years with theta-l hydrostatic dycore and SL transport and made 5-year climatologies. With diagnostics package, we compared climatologies of the new run to the default (preqx dycore, Eulerian transport, default E3SM settings) run from the same compset. Results are available here:

(latlon plots, all seasons)

https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/onguba/theta/theta.freeze.36292-nuq.allseasons/viewer/latitude-longitude/index.html

(zonal plots, all seasons)

https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/onguba/theta/theta.freeze.36292-nuq.zonal/viewer/pressure-latitude/index.html .

There is a somewhat large positive temperature bias at approximately 300 to 200 mb near the poles, visible in the "T global" pressure-latitude images and "T-200mb global" latitude-longitude images. An expert should evaluate this difference.

Test for theta-l dycore and SL transport was added to nightlies with baselines.

Additions to config and namelist user_nl_cam:

#change dycore option to theta-l
./xmlchange CAM_TARGET=theta-l

#use these settings in user_nl_cam
transport_alg=12
semi_lagrange_cdr_alg=20
rsplit=1
se_ftype=4
se_nsplit=1
se_limiter_option=9
qsplit=6
semi_lagrange_nearest_point_lev = 100
semi_lagrange_hv_q = 1
hypervis_subcycle_q = 6
theta_hydrostatic_mode=.true.
tstep_type=5
cubed_sphere_map=0
theta_advect_form=1
hypervis_subcycle=2
nu_div=1.0e15

In this simulation, we applied hyperviscosity (HV) to specific humidity q_v, which is the first tracer. Because qsplit=6 (6 dynamics steps per tracer transport step), we run 6 HV subcycles on q (hypervis_subcycle_q = 6). HV is applied to q_v after advection and before property preservation. The reason for applying HV to q_v is as follows. SL transport substantially reduces dissipation on the tracers compared with Eulerian transport. That is because SL transport does not require HV to be stable, whereas Eulerian transport does. Removing dissipation on in particular q_v can make other parts of the simulation less robust, we have found. Thus, we provide the option to apply HV to one or more tracers. The setting semi_lagrange_hv_q = 1 applies HV to just the first tracer, q_v. In the future, we want to make EAM run well without applying HV to q_v, but doing so robustly will require more study.


Relevant timers for this theta-l+SL run, timing.36292.190625-214627, on 75 anvil nodes are as follows. Columns are timer names, number of calls, and max time.

      "a:CAM_run3"                                         105121   3753.256836
              "a:prim_step_rX"                             105121   3343.381592
                  "a:ApplyCAMForcing"                      735847      7.524807
                    "a:compute_andor_apply_rhs"           3.2e+06   1101.156738
                      "a:caar_bexchV"                     3.2e+06    337.900726
                    "a:advance_hypervis"                   630726    780.730652
                      "a:biwkdp3d_bexchV"                 1.3e+06    102.427208
                      "a:ahdp_bexchV2"                    1.3e+06    100.035515
                "a:prim_step_advec"                        105121   1439.427490
                  "a:PAT_remap"                            105121   1439.272949
                    "a:Prim_Advec_Tracers_remap_ALE"       105121   1439.108643
                      "a:ALE_RKdss_bexchV"                 105121      8.113382
                      "a:SLMM_v2x"                         105121     14.941418
                      "a:SLMM_csl"                         105121    597.916321
                      "a:advance_hypervis_scalar"          105121    168.496979
                        "a:biwksc_bexchV"                  630726     76.498428
                        "a:ah_scalar_bexchV"               630726     58.095295
                      "a:CEDR"                             105121    269.654694
                      "a:CEDR_local"                       105121    204.295685
                      "a:SL_dss"                           105121    166.422882
                        "a:SLMM_bexchV"                    105121     83.393738
              "a:vertical_remap"                           105121    265.354736
              "a:prim_run_subcyle_diags"                   105121     87.252365
              "a:prim_printstate"                            1316     54.871109


Timers for the default configuration, file preqx-default-ne30-6y-march30/run/timing.2958632.bmgt1.lcrc.anl.gov.190330-171439, on 75 anvil nodes are as follows:

      "a:CAM_run3"                                         105121  13991.816406
              "a:prim_step_rX"                             630726  13261.229492
                "a:ApplyCAMForcing"                        840968      3.469257
                "a:prim_step_dyn"                          630726   3745.105225
                    "a:U3-5stage_timestep"                 630726   1602.538330
                      "a:compute_and_apply_rhs"           3.2e+06   1593.809937
                        "a:caar_bexchV"                   3.2e+06   1155.628540
                    "a:advance_hypervis_dp"                630726   2140.412109
                      "a:biwkdp3d_bexchV"                 1.9e+06    712.871643
                      "a:ahdp_bexchV2"                    1.9e+06    936.731812
                "a:prim_step_advec"                        630726   9497.830078
                  "a:PAT_remap"                            630726   9496.788086
                    "a:prim_advec_tracers_remap_rk2"       630726   9495.913086
                      "a:precomput_divdp"                  630726     17.915401
                      "a:euler_step_0"                     630726   3421.647949
                        "a:bihmix_qminmax"                1.9e+06   4816.387207
                          "a:eus_neighbor_minmax1"         630726   1564.957153
                            "a:nmm_bexchV"                 630726   1376.029663
                          "a:nmm_bexchS_start"             630726     97.431732
                          "a:biwksc_bexchV"                630726    889.131531
                          "a:nmm_bexchS_fini"              630726     22.110565
                        "a:eus_2d_advec"                  1.9e+06   4555.705566
                          "a:eus_bexchV"                  1.9e+06   1869.862549
                      "a:euler_step_1"                     630726   1699.480957
                      "a:euler_step_2"                     630726   4257.039062
                      "a:qdp_tavg"                         630726     96.223808
              "a:vertical_remap"                           210242    498.846558
              "a:prim_run_subcyle_diags"                   210242    156.251953
              "a:prim_printstate"                            1316     68.469025

Thus, tracer transport is approximately 6.5x faster and the top-level dycore timer, CAM_run3, shows approximately 3.7x speedup.