...
- The "Site" column is the machine where the tests ran
- The "Build Name" column is the name of the build and should be in the format <acme_test_suite_name>_<branch>_<compiler>, e.g. acme_developer_master_gnu
- The Update, Configure and Build columns can be ignored for us at this time
- The "Test" column is where things get interesting for us, it is split into three sub-columns:
- Not Run - For us, this is a misnomer. We put tests in Not Run if the only thing wrong with them is a namelist diff
- Fail - This is the worst result for a test. Could mean: failed to build, failed during run, or a non-BFB change in results
- Pass - The test is in good shape
- The "Build Time" column is the time when the test finished.
All our tests go into the ACME_Latest test group. The ACME_Latest group will continue to show the most recent test results for a test on the front page dashboard until new results for that test arrive. The upside is that we have a number of tests, like acme_integration on edison, that take several days to complete. Therefore, there should always be results for edison on the dashboard, even if they are a couple of days old. The downside is that users may see results on the front page and assume that they are up to date. Please pay special attention to the "Build Time" column on the far right to see how recent a result is.
There are many clickable items in the dashboard, the most useful is usually to click the number under the "Test" column of the tests you want to see (usually failing tests are the most interesting to probe, so you'd click the number in the "Fail" sub-column). This will take you to a page with some data about the test run at the top and a list of tests that had a specific result. The most useful data at the top of the page is presented in the Out-of-the-Box (OOTB) field named "OS Version". This is another field we hijacked to display information that is more useful to us; in this case, we use it to show ." Jim Foucar configured this field to display the sha1 of the commit that was tested . Below (it is not possible to rename the field though). Below the general information is a list of tests, and again, there are many clickable items. The most useful clickable item for a test is it's Status field; click . Clicking this will take you to the TestStatus.out data for this the test, which should help to give you some clue as to what happened.
Back to the front page, all our tests go into the ACME_Latest test group. The ACME_Latest group will continue to show the most recent test results for a test on the front page dashboard until new results for that test arrive. The upside is that we have a number of tests, like acme_integration on edison, that take several days to complete. With these changes, there should always be results for edison on the dashboard, even if they are a couple days old. The downside is that users may see results on the front page and assume that they are up to date. Please pay special attention to the "Build Time" column on the far right to see how recent a result is. If you want the exact commit, click the build name and look for the OS Version field (I hijacked it to show the commit).
Test Diagnosis Walkthrough
Let's walk through a hypothetical situation. Tests are failing on redsky and you want to know what happened. You go to our dashboard and see this:
If you look at the pink circle (added in by hand for clarity), In the Test-Fail column of the redsky row (circled in pink) you can see that redsky has a single failing test, and this failing test is new as of the most recent run of the tests (the +1 in the upper corner indicates that it is new). The next logical thing to do would be to click Click the "1" in the red square, which will take you to this page:
The data in the pink square (again, added by hand here) shows the id of the commit that was tested. The next logical thing is to click id of the commit that was tested is found in the "OS Version" field (highlighted it in pink, and recall that this field has been hijacked to present the sha1 of the commit). Click the red "Failed" link to go to the TestStatus.log for this failed test. You'll see something like this:
...
In many cases the TestStatus.log file will not have adequate information to diagnose a problem. For For build and run failures, log files for the problematic test will be automatically uploaded. If If results are available you'll see a gold cube next to the top-level results for a machine/compiler/branch on the main page:
The highlighted gold square highlighted in red indicates that log files are available to download for the acme_test_only_next_gnu build. Unzipping Unzipping and untarring this file will yield a directory tree in which there is a subdirectory for each failing test that will have the relevant log files.