Interpreting test results

Owned by Robert Jacob
Last updated: May 23, 2023
3 min read
A test launched by the CIME Case Control System "create_test" script goes through all the phases of configuring, building and running a case.   Each phase is checked for a PASS/FAIL condition and the results are output in the TestStatus.log file in the case directory.   A test will not be PASS unless ALL the phases are PASS.  If even one phase is FAIL, the entire test fails.

On the command line

When create_test is invoked at the command line, a "cs.status.<testid>" script is created which will parse the TestStatus.log of each test you run and output a summary of each phase.  A typical output is below for a test run with baselines.

PASS ERS.f09_g16_g.MPASLISIA.melvin_gnu CREATE_NEWCASE
PASS ERS.f09_g16_g.MPASLISIA.melvin_gnu SETUP
PASS ERS.f09_g16_g.MPASLISIA.melvin_gnu SHAREDLIB_BUILD time=164
PASS ERS.f09_g16_g.MPASLISIA.melvin_gnu NLCOMP
PASS ERS.f09_g16_g.MPASLISIA.melvin_gnu MODEL_BUILD time=95
PASS ERS.f09_g16_g.MPASLISIA.melvin_gnu SUBMIT
PASS ERS.f09_g16_g.MPASLISIA.melvin_gnu RUN time=17
PASS ERS.f09_g16_g.MPASLISIA.melvin_gnu COMPARE_base_rest
PASS ERS.f09_g16_g.MPASLISIA.melvin_gnu BASELINE
PASS ERS.f09_g16_g.MPASLISIA.melvin_gnu MEMCOMP
PASS ERS.f09_g16_g.MPASLISIA.melvin_gnu TPUTCOMP
PASS ERS.f09_g16_g.MPASLISIA.melvin_gnu MEMLEAK

See table below for more information on what a fail in a particular phase could mean.

On cdash

Within the E3SM nightly test system, the TestStatus.log is parsed in order to populate the cdash display with pass/fail and also uploaded to the dashboard for examination.  For a fail, on the cdash dashboard the Status for the test will be "Failed" in white letters over  red background.  Some fail modes are indicated by other information on the dashboard as explained below.  But you will usually have to examine the TestStatus.log file to see why a test failed.  Click on the testname on cdash to see the TestStatus.log


Diagnosing test fails

You can find how exactly test failed by searching TestStatus.log for various fail messages summarized below.  Usually you can just jump to the end of the file to see the message but sometimes the way parallel programs write to text to output files will have the fail message somewhere else in the file.

Within the test itself

While running an ERS or SMS or PET test, the test itself has many possible fail modes.  

FAIL message

FAIL meaningSettingControl
ERROR: BUILD FAIL: build e3sm failed, cat 'filename'
MODEL_BUILD FAILED for test 'testname'
Model failed to build.  Could be a code error, or compiler error or compiler license may not be available at that time.  Could run out of disk to store build artifacts.  Linking error also possible.Always onNone
RUN FAILED for test 'testname'
The "run phase" of the test failed.  This almost always means the executable exited early for some reason.   It could be a code error or a read/write problem.  A run fail is also triggered if data needed by check_input_data did not all download for any reason.Always onNone

memleak detected, memory went from X to Y in Z days
(note that this will cause the test to fail even though the word "FAIL" does not appear in the message".

The driver computes a highwater memory mark and outputs it to the coupler log each simulated day.  The log is parsed and if the highwater memory increased by more then TEST_MEMLEAK_TOLERANCE the test will fail.  Only possible if case is set to run for multiple days.Always onTurn off by adding --ignore-memleak to create_test.  TEST_MEMLEAK_TOLERANCE is defined per machine in config_machines.xml.  Can be changed in testcase directory with xmlchange.

When comparing with baselines

Every test can be given additional phases where the test results are compared with a baseline stored on the system.  If any of those additional phases FAIL, the test will FAIL.