This page should describe Validation Tests performed for this stand alone feature and should provide links to all the result pages.
...
We ran FC5AV1C-L for 6 years with preqx dycore and SL transport and made 5-year climatologies. With diagnostics package, we compared climatologies of the new run to the default (preqx dycore, Eulerian transport, default E3SM settings) run from the same compset. Results are in https://portal.nersc.gov/project/acme/onguba/sl/sl-fastest.2369911-latlon-vs-f2/viewer/latitude-longitude/index.html for latlon plots and in https://portal.nersc.gov/project/acme/onguba/sl/sl-fastest.2369911-zonal-vs-f2/viewer/pressure-latitude/index.html for zonal plots. Our understanding from communicating with C. Golaz is that differences are relatively minor and are expected.
This task required new coupling mechanisms between homme and physics (ftypes). Tests for all coupling schemes are now in nightlies with baselines.
Validation Test 2
Validation Test 2: theta-l dycore and SL transport for 1 degree
Date last modified: June 2019
Contributors:
Provenance: commit a0ff14648bf1f, A. Bradley's fork (PR #3017, now in master)
Results: /wiki/spaces/COM/pages/997752860 , Run6 only
...
There are some differences, in particular, for seasonal jets ("U global" pressure-latitude images) and for tropopause T200mb ("T global" pressure-latitude images and "T-200mb global" latitude-longitude images). An expert should evaluate these. The T200mb difference is a positive temperature bias at approximately 300 to 200 mb near the poles. There is a corresponding negative bias in relative humidity in these same regions.
...
In this simulation, we applied hyperviscosity (HV) to specific humidity q_v, which is the first tracer. Because qsplit=6 (6 dynamics steps per tracer transport step), we run 6 HV subcycles on q (hypervis_subcycle_q = 6). HV is applied to q_v after advection and before property preservation. The reason for applying HV to q_v is as follows. SL transport substantially reduces dissipation on the tracers compared with Eulerian transport. That is because SL transport does not require HV to be stable, whereas Eulerian transport does. Removing dissipation on in particular q_v can make other parts of the simulation less robust, we have found. Thus, we provide the option to apply HV to one or more tracers. The setting semi_lagrange_hv_q = 1 applies HV to just the first tracer, q_v. In the future, we want to make EAM run well without applying HV to q_v, but doing so robustly will require more study.
Timers Relevant timers for this theta-l+SL run6 as aboverun, timing.36292.190625-214627, on 75 anvil nodes :
...
are as follows. Columns are timer names, number of calls, and max time.
| Code Block |
|---|
"a:CAM_run3" 105121 3753.256836 - 105121 "a:prim_step_rX" - 3753.256836 0.670473 0.031718 0.008622 105121 3343.381592 * "a:stepon_run3" "a:ApplyCAMForcing" 735847 7.524807 - 105121 - 3753.011719"a:compute_andor_apply_rhs" 0.670471 0.0317163.2e+06 1101.156738 0.008622 "a:dyncaar_runbexchV" 3.2e+06 337.900726 "a:advance_hypervis" - 105121 - 3752.847900630726 0780.670469 730652 0.031714 0.008622 "a:primbiwkdp3d_run_sybcyclebexchV" 1.3e+06 102.427208 -"a:ahdp_bexchV2" 105121 - 3752.439697 0.670467 01.0317113e+06 100.035515 0.008622 "a:prim_diagstep_scalarsadvec" 105121 1439.427490 - 3948"a:PAT_remap" - 0.367501 0.000274 0.000082 105121 0.000324 1439.272949 "a:prim_energy_halftimesPrim_Advec_Tracers_remap_ALE" 105121 1439.108643 "a:ALE_RKdss_bexchV" - 3948 - 105121 0.424613 08.000275113382 0.000097 0.000324 "a:SLMM_v2x" "a:prim_step_rX" 105121 14.941418 -"a:SLMM_csl" 105121 - 3343.381592 0.570245 0.028336 105121 0.008622597.916321 "a:primadvance_stephypervis_dynscalar" 105121 168.496979 "a:biwksc_bexchV" - 105121 - 1899.430054 630726 0.557221 076.015519498428 0.008622 * "a:ApplyCAMForcing"ah_scalar_bexchV" 630726 58.095295 "a:CEDR" - 735847 - 7.524807 0.000213 0.000000 105121 0.060351269.654694 "a:primCEDR_advance_explocal" 105121 204.295685 - 630726 - 1892.292236 "a:SL_dss" 0.250781 0.002458 0.051730 105121 166.422882 "a:compute_andor_apply_rhs" "a:SLMM_bexchV" - 3.2e+06 - 1101.156738105121 083.204414393738 0.000266 0.258648 "a:vertical_remap" "a:caar_bexchV" 105121 265.354736 "a:prim_run_subcyle_diags" - 3.2e+06 105121 - 33787.900726252365 0.204165 0.000028 0.258648 "a:advanceprim_hypervisprintstate" 1316 - 630726 - 780.730652 0.191143 0.001057 0.051730 "a:biwkdp3d_bexchV" - 1.3e+06 - 102.427208 0.063015 0.000029 0.103459 "a:ahdp_bexchV2" - 1.3e+06 - 100.035515 0.189795 0.000029 0.103459 "a:prim_step_advec" - 105121 - 1439.427490 0.094104 0.010866 0.008622 "a:PAT_remap" - 105121 - 1439.272949 0.094103 0.010865 0.008622 "a:Prim_Advec_Tracers_remap_ALE" - 105121 - 1439.108643 0.094101 0.010863 0.008622 "a:ALE_RKdss_bexchV" - 105121 - 8.113382 0.012720 0.000024 0.008622 "a:SLMM_v2x" - 105121 - 14.941418 0.009275 0.000139 0.008622 "a:SLMM_csl" - 105121 - 597.916321 0.023259 0.004007 0.008622 "a:advance_hypervis_scalar" - 105121 - 168.496979 0.062068 0.000592 0.008622 "a:biwksc_bexchV" - 630726 - 76.498428 0.060252 0.000015 0.051730 "a:ah_scalar_bexchV" - 630726 - 58.095295 0.052509 0.000013 0.051730 "a:CEDR" - 105121 - 269.654694 0.061367 0.001182 0.008622 "a:CEDR_local" - 105121 - 204.295685 0.012888 0.001763 0.008622 "a:SL_dss" - 105121 - 166.422882 0.015599 0.000725 0.008622 "a:SLMM_bexchV" - 105121 - 83.393738 0.014745 0.000263 0.008622 "a:vertical_remap" - 105121 - 265.354736 0.016067 0.002509 0.008622 "a:vertical_remap1_1" - 210242 - 31.489454 0.008730 0.000145 0.017243 * "a:remap_Q_ppm" - 420484 - 251.081009 0.012436 0.000144 0.034486 "a:vertical_remap1_3" - 210242 - 219.885269 0.012440 0.001039 0.017243 "a:prim_run_subcyle_diags" - 105121 - 87.252365 0.009448 0.000329 0.008622 "a:prim_printstate" - 1316 - 54.871109 0.636987 0.034739 0.000108 * "a:shr_reprosum_INF_NaN_Chk" - 649945 - 0.357572 0.000195 0.000000 0.053306 * "a:shr_reprosum_int" - 649945 - 212.999924 0.395460 0.000052 0.053306 "a:repro_sum_loopa" - 649945 - 0.157211 0.000188 0.000000 0.053306 "a:repro_sum_allr_minmax" - 649945 - 176.212830 0.392793 0.000017 0.053306 "a:repro_sum_loopb" - 649945 - 0.374963 0.000528 0.000000 0.053306 "a:repro_sum_allr_i8" - 649945 - 34.479546 0.020157 0.000017 0.053306 |
Timers for default configuration, file preqx-default-ne30-6y-march30/run/timing.2958632.bmgt1.lcrc.anl.gov.190330-171439 , 75 anvil nodes:
| Code Block |
|---|
"a:CAM_run3" - 105121 - 13991.816406 1.466136 0.121596 0.008822 "a:stepon_run3" - 105121 - 13991.585938 1.466134 0.121594 0.008822 "a:dyn_run" - 105121 - 13991.424805 1.466133 0.121592 0.008822 "a:prim_run_sybcycle" - 210242 - 13990.814453 1.402953 0.055132 0.017644 "a:prim_diag_scalars" - 3948 - 0.346932 0.000193 0.000082 0.000331 "a:prim_energy_halftimes" - 3948 - 0.047745 0.000072 0.000009 0.000331 "a:prim_step_rX" - 630726 - 13261.229492 0.714745 0.014753 0.052933 * "a:ApplyCAMForcing" - 840968 - 3.469257 0.000275 0.000000 0.070577 "a:prim_step_dyn" - 630726 - 3745.105225 0.680631 0.001945 0.052933 "a:prim_advance_exp" - 630726 - 3744.255615 0.680631 0.001944 0.052933 "a:U3-5stage_timestep" - 630726 - 1602.538330 0.632909 0.000892 0.052933 "a:compute_and_apply_rhs" - 3.2e+06 - 1593.809937 0.533066 0.000159 0.264663 "a:caar_bexchV" - 3.2e+06 - 1155.628540 0.532931 0.000031 0.264663 "a:advance_hypervis_dp" - 630726 - 2140.412109 0.264765 0.001010 0.052933 "a:biwkdp3d_bexchV" - 1.9e+06 - 712.871643 0.159526 0.000031 0.158798 "a:ahdp_bexchV2" - 1.9e+06 - 936.731812 0.262727 0.000030 0.158798 "a:prim_step_advec" - 630726 - 9497.830078 0.708720 0.009799 0.052933 "a:PAT_remap" - 630726 - 9496.788086 0.708719 0.009798 0.052933 "a:prim_advec_tracers_remap_rk2" - 630726 - 9495.913086 0.708717 0.009796 0.052933 "a:precomput_divdp" - 630726 - 17.915401 0.011739 0.000014 0.052933 "a:euler_step_0" - 630726 - 3421.647949 0.195186 0.002283 0.052933 * "a:bihmix_qminmax" - 1.9e+06 - 4816.387207 0.454157 0.000159 0.158798 "a:eus_neighbor_minmax1" - 630726 - 1564.957153 0.092461 0.000399 0.052933 "a:nmm_bexchV" - 630726 - 1376.029663 0.092280 0.000234 0.052933 "a:nmm_bexchS_start" - 630726 - 97.431732 0.014166 0.000043 0.052933 "a:biwksc_bexchV" - 630726 - 889.131531 0.452009 0.000264 0.052933 "a:nmm_bexchS_fini" - 630726 - 22.110565 0.001370 0.000019 0.052933 * "a:eus_2d_advec" - 1.9e+06 - 4555.705566 0.271652 0.001314 0.158798 "a:eus_bexchV" - 1.9e+06 - 1869.862549 0.270305 0.000176 0.158798 "a:euler_step_1" - 630726 - 1699.480957 0.112606 0.001501 0.052933 "a:euler_step_2" 54.871109 |
Timers for the default configuration, file preqx-default-ne30-6y-march30/run/timing.2958632.bmgt1.lcrc.anl.gov.190330-171439, on 75 anvil nodes are as follows:
| Code Block |
|---|
"a:CAM_run3" - 630726 - 4257.039062 0.567383 105121 013991.004107816406 0.052933 "a:prim_step_rX" "a:qdp_tavg" 630726 13261.229492 * "a:ApplyCAMForcing" - 630726 - 96.223808 840968 0.001547 0.0001093.469257 0.052933 "a:verticalprim_step_remapdyn" 630726 3745.105225 - 210242 "a:U3-5stage_timestep" 498.846558 0.020524 0.002339630726 1602.538330 0.017644 "a:verticalcompute_and_remap1apply_1rhs" 3.2e+06 1593.809937 -"a:caar_bexchV" 420484 - 23.661549 0.000682 3.2e+06 01155.000051628540 0.035288 * "a:remapadvance_Qhypervis_ppmdp" 630726 2140.412109 "a:biwkdp3d_bexchV" - 1.3e+06 - 490.634094 01.0159109e+06 0712.000051871643 0.105865 "a:verticalahdp_remap1_2bexchV2" 1.9e+06 936.731812 - 420484"a:prim_step_advec" - 31.674000 0.001132 0.000073 630726 9497.830078 0.035288 "a:vertical_remap1PAT_3remap" 630726 9496.788086 - 420484 - 435.976318"a:prim_advec_tracers_remap_rk2" 0.015912 630726 09495.001023913086 0.035288 "a:prim_run_subcyle_diags"precomput_divdp" 630726 17.915401 - "a:euler_step_0" 210242 - 156.251953 0.011765 630726 03421.000629647949 * 0.017644 "a:primbihmix_printstateqminmax" 1.9e+06 4816.387207 - "a:eus_neighbor_minmax1" 1316 - 630726 68.4690251564.957153 0.414552 0.040376 0.000110 * "a:nmm_bexchV" "a:shr_reprosum_INF_NaN_Chk" 630726 1376.029663 - 655209 -"a:nmm_bexchS_start" 0.331985 0.000097630726 097.000000431732 0.054987 * "a:shrbiwksc_reprosum_intbexchV" 630726 889.131531 - 655209 - "a:nmm_bexchS_fini" 775.321594 0.636040 0.000050630726 22.110565 * 0.054987 "a:reproeus_sum2d_loopaadvec" 1.9e+06 4555.705566 - 655209 - "a:eus_bexchV" 0.184260 0.000107 0.000000 1.9e+06 0.0549871869.862549 "a:reproeuler_sumstep_allr_minmax1" 630726 1699.480957 - 655209 -"a:euler_step_2" 734.837402 0.635987 0.000016 630726 0.0549874257.039062 "a:reproqdp_sum_loopbtavg" 630726 96.223808 - 655209 "a:vertical_remap" - 0.354224 0.000197 0.000000 210242 0.054987 498.846558 "a:reproprim_sumrun_allrsubcyle_i8diags" 210242 156.251953 "a:prim_printstate" - 655209 - 38.196106 0.031034 0.000017 1316 0.05498768.469025 |
Thus, tracer transport is approximately 6.5x faster and the top-level dycore timer, CAM_run3, shows approximately 3.7x speedup.