Briefly, PASS means the test passed, FAIL means that it failed (and SFAIL, CFAIL, BFAIL, etc are all descriptions of failure in the script generation, configuration, and baseline generation stages), RUN means the test is running, and PEND means the test is waiting to be run.
The following table, assembled from the Automated Test System slides presented by CSEG, describes the result codes of tests that have passed, failed, or are still running.
| Result Code | Meaning | Where to Look First |
|---|---|---|
| Good results | ||
| PASS | Test passed | |
| Bad results | ||
| BFAIL | Compare test could not find base result | |
| CHECK | Manual review of data required | |
| ERROR | test checker failed. Test may or may not have passed. | |
| EXPFAIL | expected failure | |
| TFAIL | Test setup error | Output from create_test script (stdout) |
| SFAIL | Generation of test failed in scripts | Output from create_test script (stdout) |
| CFAIL | check_case script failed. Env variable or Build error | Output from *.test_build script (will generally point you to a build log file) |
| RUN | Run timed out or exited abnormally | (1) Batch log files in case directory: determine if it simply ran out of wall-clock time (2) TestStatus.out file in case directory (3) Log files and core files in run directory Note: Some tests (e.g. ERI) create multiple run directories, with .ref1, .ref2 extensions; you may need to check all of them, e.g. check: ERI.f10_f10.ICLM45BGC.yellowstone_intel.t01.ref1/run |
| FAIL | Test failed (either due to run failure or, e.g., no-exact restarts for an ERS test) | (1) TestStatus.out file in case directory: this will help you see the cause of failure – e.g., run didn’t complete vs. test requirements (such as exact restart) weren’t met. Look for FAIL in this file, and any messages above the FAIL line. (2) If run didn’t complete, check log files and core files in run directory (3) If requirements of test weren’t met,TestStatus.out will generally refer to differences in coupler log files and/or coupler history files. History file differences can be seen in the cprnc.out file in the run directory. |
| Test not yet complete | ||
| GEN | Test has been generated | |
| BUILD | Build succeeded, not yet submitted | |
| PEND | Test submitted, waiting in queue | |
| RUN | Test is currently running | (Check queues or log files to see whether still running or run failed) |