A test launched by the CIME Case Control System "create_test" script goes through all the phases of configuring, building and running a case. Each phase is checked for a PASS/FAIL condition and the results are output in the TestStatus.log file in the case directory. Within the E3SM nightly test system, the TestStatus.log for each test is uploaded to cdash for examination.
...
If a test end result is FAIL (on the cdash dashboard the Status for the test will be "Failed" in white letters over red background) you will have to look at the TestStatus.log output to determine how it failed (click on the testname on cdash to see the TestStatus.log).
Within the test itself
While running an ERS or SMS or PET test, the test itself has many possible fail modes.
FAIL message | FAIL meaning | Setting | Control |
|---|---|---|---|
ERROR: BUILD FAIL: build e3sm failed, cat 'filename'MODEL_BUILD FAILED for test 'testname' | Model failed to build. Could be a code error, or compiler error or compiler license may not be available at that time. Could run out of disk to store build artifacts. Linking error also possible. | Always on | Always on |
RUN FAILED for test 'testname' | The "run phase" of the test failed. This almost always means the executable exited early for some reason. It could be a code error or a rea/write problem. It could also mean that data needed by check_input_data did not all download. | Always on | Always on |
| The driver computes a highwater memory mark and outputs it to the coupler log each simulated day. The log is parsed and if the highwater memory increased by more then TEST_MEMLEAK_TOLERANCE |
| the test will fail. | Always on | Turn off by adding --ignore-memleak to create_test. TEST_MEMLEAK_TOLERANCE is defined per machine in config_machines.xml. Can be changed in testcase directory with xmlchange. |
When comparing with baselines
Every test can be given additional phases where the test results are compared with a baseline stored on the system. If any of those additional phases FAIL, the test will FAIL.