Requiring model changes to pass stringent tests before being accepted as part of E3SM’s main development branch is critical for quickly and efficiently producing a trustworthy model. Depending on their impacts on model output, code modifications can be classified into three types:
- Technical changes that continue to produce bit-for-bit identical solutions
- Changes that cause the model solution to differ, yet produce a statistically identical climate when averaged over a sufficiently long time
- Changes that lead to a different model climate
Only (3) impacts model climate, and changes of this type should only be implemented within the code after an in-depth demonstration of improvement. However, distinguishing between (2) and (3) requires a comprehensive analysis of both a baseline climate and the currently produced climate.
The MVK, PGN, and TSC tests, contained in the e3sm_atm_nbfb test suite, are used determine whether or not non-bit-for-bit (nb4b) model changes are also climate changing. The e3sm_atm_nbfb test suite is currently run nightly on NERSC's Cori and report to CdASH (https://my.cdash.org/index.php?project=ACME_Climate) under the E3SM_Custom_Tests section.
Interpreting the test results
Breifly, a PASS for these tests indicate either the tests are bit-for-bit (#1 above) or the non-bit-for-bit changes produce a statistically identical climate (#2 above), while a FAIL indicates that the non-bit-for-bit changes produce a statistically different climate(#3 above).
When used in conjuctions with the e3sm_integration test suite, the three changes can be classified into each test type above as shown in this table:
...