Running the tests

Owned by Robert Jacob
Last updated: Apr 07, 2023
3 min read

Step-by-step guide to running the tests

Questions?  Ask them in Slack: Chat for E3SM

Choose the test category you want to run, in most cases, it will be one of the e3sm-supported test categories

    • e3sm_integration test suite - A substantial set of tests run nightly on "next" and "master" on a few major platforms or workstations that establishes the correctness of the E3SM code base. Used for verifying merged code. The set of target machines should cover all supported compilers.
    • e3sm_developer test suite - A more minimal set of tests that can be run by developers on all supported platforms to instill a modicum of confidence that a set of changes does not break E3SM before issuing a PR, without incurring the computational cost of running e3sm_integration everywhere.   e3sm_developer is included within e3sm_integration.  On some machines, we only run e3sm_developer nightly because it takes less time.
    • component level suite.  Some components have defined their own tests suites.  e.g. e3sm_land_developer, e3sm_atm_developer,  e3sm_mosart_developer.

To run a given test suite on a given machine, use the create_test command, issued from the cime/scripts directory in your E3SM source tree:

    1. cd E3SM/cime/scripts
    2. ./create_test test_suite  [--compiler compiler --baseline-root e3sm_baseline_dir -t test_id -r test_root_dir ] [-(c|g) [-b baseline_name ]] [--project project_or_account ]
      1. machine - The E3SM scripts will probe your machine name from the current nodename
      2. compiler - The compiler toolset you wish to use, examples are: gnu, intel, pgi. You do not need to provide this if you want the default compiler (the first compiler listed in the COMPILERS parameter for your machine)
      3. test_suite - The name of the test category you want to run, like e3sm_developer. You can also provide a list of test names.  Instead a test suite, you can specify one specific test or set of tests.
      4. test_root_dir - The path where your test cases will be dumped. Default will be CIME_OUTPUT_ROOT for the machine you are using.
      5. e3sm_baseline_dir - You only need to specify this if you want to use a different baseline area than is specified for your machine by the BASELINE_ROOT parameter
      6. -c compare against baselines you generated in Generating new baseline files for a topic branch or the system default baselines which are usually from latest master.
      7. If you generate new baselines, especially for shared master and next branches, make them group-writable by others:
          1. chmod -R g+w $BASELINE_ROOT/$branch_name
      8. baseline_subdir - The name of the baselines (usually the major release you're on) you want to use. The default will be <current-branch-name>. Note that the tool will always prepend the compiler to the baseline subdir. Our baselines are always segregated by compiler. Don't bother with this option unless you provided -c or -g.
      9. project_or_account - The id that lets you run batch jobs on this machine
      10. test_id  - a string appended to end of test name.  default is a date and sha such as ".20210708_184630_c657sy"
      11. for a complete list of options, do "./create_test --help"
    3. Once the tests are running, you'll want to see test results
      1. Case 1: Simple
        1. cd test_root_dir
        2. you will find a script named cs.status.(testid) You can run this script from the test root directory to see the status of tests being run within this suite
      2. Case 2: We have a more sophisticated script called wait_for_tests that provides extra capabilites like: waiting for tests to finish, converting results to CTest and submitting to a CDash dashboard
        1. cd test_root_dir
        2. <E3SM>/cime/CIME/wait_for_tests [ -d buildname] */TestStatus
    4. Example 1  Testing without baselines: to run the 'e3sm_developer' suite on Chrysalis using the Intel compiler, you would run:
      1. (make sure the clone is checked out on your development branch)
      2. cd E3SM/cime/scripts
      3. ./create_test e3sm_developer -r mytestrun (the script will detect its on Chrysalis and use the default compiler, intel)
      4. cd /lcrc/group/e3sm/$USERID/scratch/chrys/mytestrun
      5. ./cs.status.(testid).
    5. Example 2:  Testing against baselines:

See Interpreting test results.

See the CSEG presentation on testing for more information on the testing system.





Filter by label

There are no items with the selected labels at this time.