Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 9 Next »

This page documents how to configure, build and run standalone HOMME (EAM dynamical core) for idealized test cases on a supported platform.

Directories: (set these variables based on your configuration/preferences and target machine):

e3sm = /path/to/github/code/checkout/E3SM

homme = $e3sm/components/homme
wdir = /path/to/working/directory/for/compiling/and/running
mach = $homme/cmake/machineFiles/cori-knl.cmake (see directory for other supported machines)

  1. Load the same modules that a full E3SM case would load: (put in your .login or .bashrc )

    1. E3SM’s CIME provides a script which will give the csh output needed to load the correct modules and other configuration settings:

    2. eval `$e3sm/cime/scripts/Tools/get_case_env`

    3. Most users will be using the bash shell, in which case examine the output of $e3sm/cime/script/Tools/get_case_env and modify to work with bash

  2. Run cmake from your $wdir. (only done once for each git checkout)

    1. cd $wdir
cmake -C  $mach $homme
make -j4 theta-l   # optional step to see if code will compile

    2. Should only need to be redone if source code dependencies or compiler options change. To rerun from an existing $wdir (keeping the cmake cache file), run “cmake .”

    3. For additional CMAKE options, see $homme/README.cmake. Options include number of vertical levels (default 26), type of output, native or lat/lon or disabled, ability to enable/disable energy diagnostics, ability to disable kokkos components (SL transport).

  3. After step 2 is complete, one can build many different targets. The generic targets are:

    1. preqx, preqx_kokkos. E3SM v1 hydrostatic dycore (Fortran and C++ version)

    2. theta-l, theta-l_kokkos. E3SM v2 NH dycore

    3. sweqx. shallow water version

    4. Then there are a host of specific targets such as theta-l-nlev20-native that are used by various test cases.

  4. Run DCMIP2012 Test 3-1 (simple gravity wave)

    1. cd $wdir/dcmip_tests/dcmip2012_test3.1_nh_gravity_waves/theta-l
make install # Copies test case related files from $homme
./build.sh # Build the correct executable with correct compile time options for this test

    2. Look for a suitable jobscript for your machine with ls jobscript-sh*. Most test cases include jobscript-snl.sh which runs on E3SM institutional clusters, and jobscript-knl.sh for NERSC Cori. Some editing may be necessary.

      1. The jobscript usually runs included NCL scripts for creating standard test case plots from the output. This may require the user to load an ncl module.

  5. All the DCMIP2012 and DCMIP2016 test cases can be run similar to the above example for Test 3-1.

  6. Other standalone HOMME test cases of interest. These usually contain a single script that will configure and build the model (assuming step1 above is done)

    1. Held-Suarez: see $homme/test/held_suarez0 and the script hs.job.

    2. NGGPS and V1 benchmarks: see $homme/test/benchmark and associated scripts.

  7. HOMME doubly periodic test cases

Example Scripts (Cori-KNL)

Here we include two example Bash scripts, setup-homme.sh and jobscript-knl.sh, which execute the instructions above and submit a job to execute Test 3-1 (simple gravity wave).

setup-homme.sh
#!/bin/bash

# Sets file paths for running the dynamical core (HOMME) of the E3SM

echo '-- Setting file paths...'

e3sm=/global/homes/j/jlturner/practicum-2021/E3SM
homme=$e3sm/components/homme
wdir=/global/cscratch1/sd/jlturner/practicum-2021/homme
mach=$homme/cmake/machineFiles/cori-knl.cmake
test3_1=$wdir/dcmip_tests/dcmip2012_test3.1_nh_gravity_waves/theta-l

export e3sm
export homme
export wdir
export mach
export test3_1

# Load the necessary modules

echo '-- Loading necessary modules...'

eval `$e3sm/cime/scripts/Tools/get_case_env`

# Compile HOMME

cwd=$(pwd)
echo '-- Compiling HOMME...'

cd $wdir
cmake -C $mach $homme
make -j4 theta-l

cd $cwd

# Compile Test 3.1

cd $test3_1
make install
./build.sh

Note: The setup can be a little finicky if you have been doing other work during your current session. It is recommended that you run this script from a ‘fresh’ session (i.e., logout and login).

jobscript-knl.sh
#!/bin/bash 
#
#   Jobscript for launching dcmip2012 test3-1 on Cori KNL
#
#SBATCH --job-name=d31-theta
#SBATCH --output=%j.out
#SBATCH --error=%j.err

#SBATCH --qos=debug
#SBATCH --nodes=4
#SBATCH --constraint=knl

#SBATCH --account=e3sm
#SBATCH --mail-user=jason.torchinsky@wisc.edu
#SBATCH --mail-type=all

#SBATCH --time=0-00:30:00

# Set variables, load modules
OMP_NUM_THREADS=1
EXEC=../../../test_execs/theta-l-nlev20/theta-l-nlev20
module load ncl

#############################################################################
# theta (hydrostatic)
#############################################################################
namelist=./namelist-h.nl
cp -f $namelist input.nl
srun -K -c 1 -N $SLURM_NNODES  $EXEC < input.nl

ncl plot_omega.ncl
ncl plot_theta.ncl

mv -f test31_omega.pdf                 hydro_test31_omega.pdf
mv -f dcmip2012_test3_theta_diff.pdf   hydro_test3_theta_diff.pdf
mv -f dcmip2012_test3_theta_diff_last.pdf   hydro_test3_theta_diff_last.pdf
mv -f movies/dcmip2012_test31.nc       movies/hydro_dcmip2012_test31.nc

#############################################################################
# theta-nh (non-hydrostatic)
#############################################################################
namelist=./namelist-nh.nl
cp -f $namelist input.nl
srun -K -c 1 -N $SLURM_NNODES  $EXEC < input.nl

ncl plot_omega.ncl
ncl plot_theta.ncl

mv -f test31_omega.pdf                 nonhydro_test31_omega.pdf
mv -f dcmip2012_test3_theta_diff.pdf   nonhydro_test3_theta_diff.pdf  
mv -f dcmip2012_test3_theta_diff_last.pdf   nonhydro_test3_theta_diff_last.pdf  
mv -f movies/dcmip2012_test31.nc        movies/nonhydro_dcmip2012_test31.nc

Note: The time limit on jobscript-knl.sh may need to be adjusted (2021/05/12).

  • No labels