Page Comparison

...

• MVK: The mutivariate Kolmogorov-Smirnov Test     POC Salil Mahajan
  
  This tests the null hypothesis that the reference (n) and modified (m) model Short Independent Simulation Ensembles (SISE) represent the same climate state, based on the equality of distribution of each variable's annual global average in the standard monthly model output between the two simulations.
  
  The (per variable) null hypothesis uses the non-parametric, two-sample (n and m) Kolmogorov-Smirnov test as the univariate test of equality of distribution of global means. The test statistic (t) is the number of variables that reject the (per variable) null hypothesis of equality of distribution at a 95% confidence level. The (overall) null hypothesis is rejected if t > α, where α is some critical number of rejecting variables. The critical value, α, is obtained from an empirically derived approximate null distribution of t using resampling techniques.
  
  For more information, see:
  • Salil Mahajan, Katherine J Evans, Joseph H Kennedy, Min Xu, Mathew R Norman, and Marcia L Branstetter. Ongoing solution reproducibility of earth system models as they progress toward exascale computing. The International Journal of High Performance Computing Applications, 0(0):1094342019837341, 0. doi:10.1177/1094342019837341.
    
  • Salil Mahajan, Abigail L. Gaddis, Katherine J. Evans, and Matthew R. Norman. Exploring an ensemble-based approach to atmospheric climate modeling and testing at scale. Procedia Computer Science, 108:735 – 744, 2017. International Conference on Computational Science, ICCS 2017, 12-14 June 2017, Zurich, Switzerland. doi:10.1016/j.procs.2017.05.259.
    
    
• PGN: The perterbation Growth Test   POC  Balwinder Singh
  
  This tests the null hypothesis that the reference (n) and modified (m) model ensembles represent the same atmospheric state after each physics parameterization is applied within a single time-step using the two-sample (n and m) T-test for equal averages at a 95% confidence level. Ensembles are generated by repeating the simulation for many initial conditions, with each initial condition subject to multiple perturbations.
  
  For more information, see:
  • Singh, P. J. Rasch, H. Wan, W. Ma, P. H. Worley, and J. Edwards. A verification strategy for atmospheric model codes using initial condition perturbations. Journal of Geophysical Research: Atmospheres, In prep.
    
    
• TSC: The Time Step Convergence Test   POC Hui Wan
  
  This tests the null hypothesis that the convergence of the time stepping error for a set of key atmospheric variables is the same for a reference ensemble and a test ensemble. Both the reference and test ensemble are generated with a two-second time step, and for each variable the RMSD between each ensemble and a truth ensemble, generated with a one-second time step, is calculated. RMSD is calculated globally and over two domains, the land and the ocean. The land/ocean domains contain just the atmosphere points that are over land/ocean cells.
  
  At each 10 second interval during the 10 minute long simulations, the difference in the reference and test RMSDs for each variable, each ensemble member, and each domain are calculated and these ΔRMSDs should be zero for identical climates. A one sided (due to self convergence) Student's T Test is used to test the null hypothesis that the ensemble mean ΔRMSD is statistically zero at the 99.5% confidence level. A rejection of the null hypothesis (mean ΔRMSD is not statistically zero) at any time step for any variable will cause this test to fail.
  
  For more information, see:
  • H. Wan, K. Zhang, P. J. Rasch, B. Singh, X. Chen, and J. Edwards. A new and inexpensive non-bit-for-bit solution reproducibility test based on time step convergence (tsc1.0). Geoscientific Model Development, 10(2):537–552, 2017. doi:10.5194/gmd-10-537-2017.

Interpreting the test results

...

Unlike the other E3SM/CIME system tests, the e3sm_atm_nb4b test must generate a new baseline because they require more than the final history and restart files to perform their statistical analyses (can't just copy files like we normally do when blessing a failed test.

For `cori-knl`, this would look like: anvil, the following would generate (-g) and overwrite (-o)  all three test baselines:

# export E3SM=/PATH/TO/E3SM
cd ${E3SM}/cime/scripts
source /globallcrc/cfssoft/cdirsclimate/e3sm/software/anaconda_envs-unified/load_latest_cime_env.sh
./create_test e3sm_atm_nbfb -g -o --walltime 0200:0020:00

This process will need to be repeated for every machine the test are being run on (currently only cori-knl). For more detailed information on how to run these tests, see manually running the tests section below.To generate a new individual baseline it is the same procedure:

# export E3SM=/PATH/TO/E3SM
cd ${E3SM}/cime/scripts
source /lcrc/soft/climate/e3sm-unified/load_latest_cime_env.sh
./create_test MVK_PS.ne4_ne4.FC5AV1C-L -g -o --walltime 00:20:00

This process will need to be repeated for every machine the test are being run on (currently only anvil). For more detailed information on how to run these tests, see manually running the tests section below.

For cori-knl the procedure is similar, but with increased walltime and a different path for the CIME environment

# export E3SM=/PATH/TO/E3SM
cd ${E3SM}/cime/scripts
source /global/cfs/cdirs/e3sm/software/anaconda_envs/load_latest_cime_env.sh
./create_test e3sm_atm_nbfb -g -o --walltime 01:00:00

Manually running the tests

...

./create_test MVK_PLPS.ne4_ne4.FC5AV1C-LF2010 -g --baseline-root "/PATH/TO/BASELINE" 

...

./create_test MVK_PLPS.ne4_ne4.FC5AV1C-LF2010 -c --baseline-root "/PATH/TO/BASELINE" 

...

where <CASE_SPECIFIER> is the case string you would have passed to create_test (e.g., MVK_PLPS.ne4_ne4.FC5AV1C-L). EVV will then report to you a URL where you can view the website:

--------------------------------------------------------------------
                   ______  __      __ __      __                    
                  |  ____| \ \    / / \ \    / /                    
                  | |__     \ \  / /   \ \  / /                     
                  |  __|     \ \/ /     \ \/ /                      
                  | |____     \  /       \  /                       
                  |______|_|     \/         \/                        
           \/         \/                                                
    Extended Verification and Validation for Earth System Models    
--------------------------------------------------------------------

  Current run: YYYY-MM-DD HH:MM:SS
  User: ${USER}
  OS Type: Linux 4.12.14-150.27-default
  Machine: cori07
  

  
    Extended Verification and Validation for Earth System Models    
--------------------------------------------------------------------

  Current run: YYYY-MM-DD HH:MM:SS
  User: ${USER}
  OS Type: Linux 4.12.14-150.27-default
  Machine: cori07
  

Serving HTTP on 0.0.0.0 port 8080 (http://0.0.0.0:8080/)

View the generated website by navigating to:

    http://0.0.0.0:8080/<CASE_SPECIFIER>L.cori-knl_intel.C.YYYYMMDD_HHMMSS_RANDOMID.evv/index.html

Exit by pressing `ctrl+c` to send a keyboard interrupt.

...

Serving HTTP on 0.0.0.0 port 8080 (http://0.0.0.0:8080/)

View the generated website by navigating to:

    http://0.0.0.0:8080/<CASE_SPECIFIER>L.cori-knl_intel.C.YYYYMMDD_HHMMSS_RANDOMID.evv/index.html

Exit by pressing `ctrl+c` to send a keyboard interrupt.

You can now either click that link or copy-past it into your favorite web browser for viewing.

View output from tests run on Anvil:  

The sshd server on blues (frontend for Anvil) blocks port forwarding (as of 2022/6) so the ssh -L command above will not work. 
However, the evv output is automatically copied into the LCRC webserver at  /lcrc/group/e3sm/public_html/$USER/evv, and will be viewable (with $USER replaced by your user name) at:

https://web.lcrc.anl.gov/public/e3sm/$USER/evv/

View a local copy

For this example, we'll assume the tests were run on Cori at NERSC, but these instructions should be easily adaptable to any E3SM supported machine. First, on your local machine, install the cime_env conda env and activate it:

...

where ${CASE_RUN_DIR}is as described above and <CASE_SPECIFIER> is the case string you would have passed to create_test (e.g., MVK_PLPS.ne4_ne4.FC5AV1C-L). You can now either click that link or copy-paste that link into your favorite web browser to view the output website.

...