W10 NH Dycore with SL transport Validation Phase 1
This page should describe Validation Tests performed for this stand alone feature and should provide links to all the result pages.
Summary
We run a 1 degree FC compset, FC5AV1C-L, simulation for 6 years and create 5-year (year 2 to 6) climatologies from it. We plot climatologies vs observations and vs default configuration. The default configuration consists of default hydrostatic dycore with default settings. Preliminary runs are described in /wiki/spaces/COM/pages/780664997 (with preqx dycore) and in /wiki/spaces/COM/pages/941096965 (with theta dycore), each relevant run also has entry for its climatologies.
SL transport was first evaluated in the preqx dycore using the E3SM v1 atmosphere at 1 degree. Results and some technical details are presented in /wiki/spaces/COM/pages/916324466 . The SL transport was shown to be 6x faster than Eulerian transport.
With theta dycore, we only explore the hydrostatic option for now. The preliminary configuration for theta-l dycore and SL transport is Run6 in /wiki/spaces/COM/pages/997752860 and is summarized below. We assess the climatology differences to be small overall, but below we note below specific differences that an expert should evaluate. In this validation test, the transport speedup is over 6x and the overall dycore speedup is approximately 3.7x for an FC compset.
Validation Test 1
Validation Test 1: preqx dycore and SL transport for 1 degree
Date last modified: Sept. 2018
Contributors:
Provenance: commit 099c4dd, A. Bradley's fork of E3SM
Results: /wiki/spaces/COM/pages/780664997 , Run 19.
We ran FC5AV1C-L for 6 years with preqx dycore and SL transport and made 5-year climatologies. With diagnostics package, we compared climatologies of the new run to the default (preqx dycore, Eulerian transport, default E3SM settings) run from the same compset. Results are in https://portal.nersc.gov/project/acme/onguba/sl/sl-fastest.2369911-latlon-vs-f2/viewer/latitude-longitude/index.html for latlon plots and in https://portal.nersc.gov/project/acme/onguba/sl/sl-fastest.2369911-zonal-vs-f2/viewer/pressure-latitude/index.html for zonal plots. Our understanding from communicating with C. Golaz is that differences are relatively minor and are expected.
This task required new coupling mechanisms between homme and physics (ftypes). Tests for all coupling schemes are now in nightlies with baselines.
Validation Test 2
Validation Test 2: theta-l dycore and SL transport for 1 degree
Date last modified: June 2019
Contributors:
Provenance: commit a0ff14648bf1f, A. Bradley's fork (PR #3017, now in master)
Results: /wiki/spaces/COM/pages/997752860 , Run6 only
We ran FC5AV1C-L for 6 years with theta-l hydrostatic dycore and SL transport and made 5-year climatologies. With diagnostics package, we compared climatologies of the new run to the default (preqx dycore, Eulerian transport, default E3SM settings) run from the same compset. Results are available here:
(latlon plots, all seasons)
(zonal plots, all seasons)
There is a somewhat large positive temperature bias at approximately 300 to 200 mb near the poles, visible in the "T global" pressure-latitude images and "T-200mb global" latitude-longitude images. An expert should evaluate this difference.
Test for theta-l dycore and SL transport was added to nightlies with baselines.
Additions to config and namelist user_nl_cam:
#change dycore option to theta-l ./xmlchange CAM_TARGET=theta-l #use these settings in user_nl_cam transport_alg=12 semi_lagrange_cdr_alg=20 rsplit=1 se_ftype=4 se_nsplit=1 se_limiter_option=9 qsplit=6 semi_lagrange_nearest_point_lev = 100 semi_lagrange_hv_q = 1 hypervis_subcycle_q = 6 theta_hydrostatic_mode=.true. tstep_type=5 cubed_sphere_map=0 theta_advect_form=1 hypervis_subcycle=2 nu_div=1.0e15
In this simulation, we applied hyperviscosity (HV) to specific humidity q_v, which is the first tracer. Because qsplit=6 (6 dynamics steps per tracer transport step), we run 6 HV subcycles on q (hypervis_subcycle_q = 6). HV is applied to q_v after advection and before property preservation. The reason for applying HV to q_v is as follows. SL transport substantially reduces dissipation on the tracers compared with Eulerian transport. That is because SL transport does not require HV to be stable, whereas Eulerian transport does. Removing dissipation on in particular q_v can make other parts of the simulation less robust, we have found. Thus, we provide the option to apply HV to one or more tracers. The setting semi_lagrange_hv_q = 1 applies HV to just the first tracer, q_v. In the future, we want to make EAM run well without applying HV to q_v, but doing so robustly will require more study.
Relevant timers for this theta-l+SL run, timing.36292.190625-214627, on 75 anvil nodes are as follows. Columns are timer names, number of calls, and max time.
"a:CAM_run3" 105121 3753.256836
"a:prim_step_rX" 105121 3343.381592
"a:ApplyCAMForcing" 735847 7.524807
"a:compute_andor_apply_rhs" 3.2e+06 1101.156738
"a:caar_bexchV" 3.2e+06 337.900726
"a:advance_hypervis" 630726 780.730652
"a:biwkdp3d_bexchV" 1.3e+06 102.427208
"a:ahdp_bexchV2" 1.3e+06 100.035515
"a:prim_step_advec" 105121 1439.427490
"a:PAT_remap" 105121 1439.272949
"a:Prim_Advec_Tracers_remap_ALE" 105121 1439.108643
"a:ALE_RKdss_bexchV" 105121 8.113382
"a:SLMM_v2x" 105121 14.941418
"a:SLMM_csl" 105121 597.916321
"a:advance_hypervis_scalar" 105121 168.496979
"a:biwksc_bexchV" 630726 76.498428
"a:ah_scalar_bexchV" 630726 58.095295
"a:CEDR" 105121 269.654694
"a:CEDR_local" 105121 204.295685
"a:SL_dss" 105121 166.422882
"a:SLMM_bexchV" 105121 83.393738
"a:vertical_remap" 105121 265.354736
"a:prim_run_subcyle_diags" 105121 87.252365
"a:prim_printstate" 1316 54.871109
Timers for the default configuration, file preqx-default-ne30-6y-march30/run/timing.2958632.bmgt1.lcrc.anl.gov.190330-171439, on 75 anvil nodes are as follows:
"a:CAM_run3" 105121 13991.816406
"a:prim_step_rX" 630726 13261.229492
"a:ApplyCAMForcing" 840968 3.469257
"a:prim_step_dyn" 630726 3745.105225
"a:U3-5stage_timestep" 630726 1602.538330
"a:compute_and_apply_rhs" 3.2e+06 1593.809937
"a:caar_bexchV" 3.2e+06 1155.628540
"a:advance_hypervis_dp" 630726 2140.412109
"a:biwkdp3d_bexchV" 1.9e+06 712.871643
"a:ahdp_bexchV2" 1.9e+06 936.731812
"a:prim_step_advec" 630726 9497.830078
"a:PAT_remap" 630726 9496.788086
"a:prim_advec_tracers_remap_rk2" 630726 9495.913086
"a:precomput_divdp" 630726 17.915401
"a:euler_step_0" 630726 3421.647949
"a:bihmix_qminmax" 1.9e+06 4816.387207
"a:eus_neighbor_minmax1" 630726 1564.957153
"a:nmm_bexchV" 630726 1376.029663
"a:nmm_bexchS_start" 630726 97.431732
"a:biwksc_bexchV" 630726 889.131531
"a:nmm_bexchS_fini" 630726 22.110565
"a:eus_2d_advec" 1.9e+06 4555.705566
"a:eus_bexchV" 1.9e+06 1869.862549
"a:euler_step_1" 630726 1699.480957
"a:euler_step_2" 630726 4257.039062
"a:qdp_tavg" 630726 96.223808
"a:vertical_remap" 210242 498.846558
"a:prim_run_subcyle_diags" 210242 156.251953
"a:prim_printstate" 1316 68.469025
Thus, tracer transport is approximately 6.5x faster and the top-level dycore timer, CAM_run3, shows approximately 3.7x speedup.