Page Comparison

Step-by-step guide to running the tests

Questions?  Ask them in /wiki/spaces/SE/pages/19464303Slack: Chat for E3SM

Choose the test category you want to run, in most cases, it will be one of the

...

e3sm-supported test categories

1. • acmee3sm_integration test suite - A substantial set of tests run nightly on "next" and "master" on a few major platforms or workstations that establishes the correctness of the ACME E3SM code base. Used for verifying merged code. The set of target machines should cover all supported compilers.acme
   • e3sm_developer test suite - A more minimal set of tests that can be run by develpers developers on all supported platforms to instill a modicum of confidence that a set of changes does not break ACME E3SM before issuing a PR, without incurring the computational cost of running acme_full or acme_integration everywhere e3sm_integration everywhere.   e3sm_developer is included within e3sm_integration.  On some machines, we only run e3sm_developer nightly because it takes less time.
   • component level suite.  Some components have defined their own tests suites.  e.g. e3sm_land_developer, e3sm_atm_developer,  e3sm_mosart_developer.

To run a given test suite on a given machine, use the create_test command, issued from the cime/scripts directory  directory in your ACME E3SM source tree:

• 1. cd ACMEE3SM/cime/scripts-acme
  2. ./create_test test_suite -testid test_label -testroot test_root_dirsuite [--compiler compiler --baseline-rootacmeroot e3sm_baseline_dir -testid t test_id -r test_labelroot_dir] [-(comparec|generateg) [-b baseline_subdirname ]] [--project project_or_account ]
     
     1. machine - The ACME name for the machine that you're on, all lowercase, look here for supported machinesE3SM scripts will probe your machine name from the current nodename
     2. compiler - The compiler toolset you wish to use, examples are: gnu, intel, pgi. You do not need to provide this if you want the default compiler (the first compiler listed in the COMPILERS parameter for your machine)
     3. test_suite - The name of the test category you want to run, like acmee3sm_developertestroot . You can also provide a list of test names.  Instead a test suite, you can specify one specific test or set of tests.
     4. test_root_dir - The path where your test cases will be dumpedacme. Default will be CIME_OUTPUT_ROOT for the machine you are using.
     5. e3sm_baseline_dir - You only need to specify this if you want to use a different baseline area than is specified in <ACME>/scripts/ccsm_utils/Machines/config_machines.xml for your machinefor your machine by the BASELINE_ROOT parameter
     6. -c compare against baselines you generated in Generating new baseline files for a topic branch or the system default baselines which are usually from latest master.
     7. If you generate new baselines, especially for shared master and next branches, make them group-writable by others:
        1. 1. chmod -R g+w $BASELINE_ROOT/$branch_name
     8. baseline_subdir - The name of the baselines (usually the major release you're on) you want to use. Use -generate if this is the first time running this test on this machine OR if you want to regenerate baseline results for this test (approved BFB change); otherwise, always use -compareThe default will be <current-branch-name>. Note that the tool will always prepend the compiler to the baseline subdir. Our baselines are always segregated by compiler. Don't bother with this option unless you provided -c or -g.
     9. project_or_account - The id that lets you run batch jobs on this machine
     10. test_id  - a string appended to end of test name.  default is a date and sha such as ".20210708_184630_c657sy"
     11. for a complete list of options, do "./create_test --help"
  3. Once the tests are running, you'll want to see test results
     1. Case 1: Simple
        
        1. cd test_root_dir
        2. you will find a script named cs.status.(testid).(machine) . You can run this script from the test root directory to see the status of tests being run within this suite
     2. Case 2: We have a more sophisticated script called wait_for_tests that provides extra capabilites like: waiting for tests to finish, converting results to CTest and submitting to a CDash dashboard
        
        1. cd test_root_dir
        2. <ACME><E3SM>/scriptscime/acmeCIME/wait_for_tests [ -d buildname] */TestStatus
  4. Example 1  Testing without baselines: to run the 'acmee3sm_developer' suite on Edison Chrysalis using the Intel compiler (assuming everything has been set up and this is not the first run), you would run:
     1. (make sure the clone is checked out on your development branch)
     2. cd ACMEE3SM/cime/scripts
     3. ./create_test -xmle3sm_mach edison -xml_compiler intel -xml_category acme_developer -testid acme_dev -testroot $SCRATCH/acme_dev -compare v0.1 -project acmecd $SCRATCH/acme_devdeveloper -r mytestrun (the script will detect its on Chrysalis and use the default compiler, intel)
     4. cd /lcrc/group/e3sm/$USERID/scratch/chrys/mytestrun
     5. ./cs.status.acme_dev.edison(testid).
  5. Example 2:  Testing against baselines:

See Interpreting test results.

See the CSEG presentation on testing for more information on the testing system.

Related articles