During the V2 development process, we transitioned from CLUBB V1 to CLUBB V2. In CLUBB V2, a process controlled by c_k10 was modified to add finer control with c_k10 and c_k10h. The old behavior was recovered if c_k10h = c_k10 = 0.35. In V2 development, c_k10h was not set in the namelist, inheriting a default value of c_k10h = 1.0, resulting in an unexpected change in CLUBB which was not detected for several months. For more detailed background, see V2 Case Studies
Can E3SM’s NBFB tests detect this difference? And how sensitive are they to changes in c_k10h?
Additional test (harder to perform, since CLUBB V1 is on in older code): Will the NBFB tests consider simulations with CLUBB V1 (c_k10=0.35) statistically similar to CLUBB V2 (c_k10h=c_k10=0.35)?
Tests were run on Compy. All tests are first run with -g to generate baselines. Then they are rerun with -c (compare to baseline), with various values of c_k10h
MVK_P24x1.ne4_oQU240.F2010-CICE
30 member ensemble of ~ 1 year simulations. Takes about 1.3 hours on 18 nodes.
c_10kh | Climate Statistically Different? | Metrics threshold=13 |
|---|---|---|
0.35 (default) | No | |
0.36 | No | reject 7/121 |
0.38 | Yes | reject 21/121 |
0.40 | Yes | reject 50/121 |
Hack to reuse same “-c” case to run multiple experiments:
rm -f run/*.nc (otherwise we get PIO run time errors)
add “clubb_c_k10h=0.40” to user_nl_eam_???? files
./case.submit
PGN_P32x1.ne4_oQU240.F2010
20 member ensemble of ~ 1 timestep simulations. Takes about 1 min on 16 nodes.
c_10kh | Climate Statistically Different? | Metrics T test (t,p) |
|---|---|---|
0.35 (default) | No | (0.000, 1.000) |
0.350000001d0 | No | (-1.424, 0.169) |
0.35000001 | Yes | (-2.542, 0.019); |
0.36 | Yes | (-12.564, 0.000); |
Hack to reuse same “-c” case to run multiple experiments:
rm -f run/*.nc (otherwise we get PIO run time errors)
add “clubb_c_k10h=0.40” to user_nl_eam_???? files
./case.submit
TSC_P36x1.ne4_ne4.F2010-CICE
12 member ensemble of 5 day simulations. Takes about 10min on 11 nodes.
Hack to reuse a “-c” case to run multiple experiments above does not work. During the RUN phase, the user_nl_eam_???? ensemble member namelists will be created anew by cime/scripts/lib/CIME/SystemTests/tsc.py.