...

Setting some aliases may be useful in running the model. You can edit ~/.bashrc to add aliases. Run source ~/.bashrc to start using them.

...

Batch jobs

To check on all running processesbatch jobs:

alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10M %.10l %.8Q %55j" --sort=P,-t,-p'

To check on your running processesbatch jobs:

alias sq='sqa -u $USER'

The output of sq uses several abbreviations: ST = Status, R = running, PD = pending, CG = completing.

...

You will be working in several directories.

On Anvil anvil or Chrysalis chrysalis (i.e., LCRC machines), those paths are:

<run_scripts_dir>: /home/<username>/${HOME}/E3SM/scripts<simulations

<code_source_dir>: ${HOME}/lcrc/group/e3sm/<username>/E3SM_simulationsE3SM/code

<post_processing_script_dir>: /home/<username>${HOME}/E3SM/utils/post_v2

<simulations_dir>: /lcrc/group/e3sm/<username>/E3SM_simulations

So, it may be useful to set the following aliases:

# Model running
alias run_scripts="cd ${HOME}/E3SM/scripts/"
alias simulations="cd /lcrc/homegroup/e3sm/<username>/E3SM/scripts_simulations/"
alias simulationspost_processing="cd /lcrc/group/e3sm/${HOME}/E3SM/utils/post_v2/"

On other machines, the paths are the same, except for the <simulations_dir>.

On compy (PNNL):

<simulations_dir>: /compyfs/<username>/E3SM_simulations

...

On cori (NERSC):

<simulations_dir>: ${CSCRATCH}/E3SM_simulations

Configuring the Model Run – Old Run Script

...

• readonly COMPSET="WCYCL1850" : compset (configuration)

• readonly RESOLUTION="ne30pg2_EC30to60E2r2-1900_ICG": resolution

  • ne30 is the number of spectral elements for the atmospheric grid., pg2 refers to the phys-grid option

  • EC30to60E2r2 is the ocean and sea-ice resolution.

  • ICG means initial conditions from a G-case. ICG can apply to either EC30to60E2r2 or RRM ocean (WC14). It specifies whether the ocean and ice is partially spun up or just from data (temperature and salinity, velocity is assumed zero): “G-case” indicates the ocean and sea-ice are active but the atmosphere, land, and river are from data.

  • rrm for regionally refined mesh is an option to replace other resolutions. RRM for the ocean/sea ice is 14km near the US coast and Arctic and identical to the 30to60E2r2 elsewhere. RRM for the atmosphere is 120km over North America and ne30 elsewhere.

• readonly DESCRIPTOR="v2_test02.piControl.ne30pg2_EC30to60E2r2":

  • v2_test02 is a short custom description to help identify the simulation.

  • piControl is the type of simulation. Other options here include amip, F2010.

  • ne30pg2_EC30to60E2r2 is the resolution. This should be identical to the RESOLUTION above.

# Code and compilation

• readonly CHECKOUT="20210205": date the code was checked out on in the form 20210205": date the code was checked out on in the form {year}{month}{day}. The source code will be checked out in sub-directory named {year}{month}{day} under <code_source_dir>.

• readonly BRANCH="master": branch the code was checked out from. Valid options include “master”, a branch name or a git hash.

• readonly DEBUG_COMPILE=false : option to compile with DEBUG flag (leave to false)

# Run options

For a short test run, this section might look like:

readonly MODEL_START_TYPE="initial"  # initial, continue
readonly START_DATE="0001-01-01"
readonly STOP_OPTION="nmonths" # Units will be number of months
readonly STOP_N="1"            # Stop after running 1 month (one `STOP_OPTION`)
readonly REST_OPTION="nmonths" # Units will be number of months
readonly REST_N="1"            # Write restart file after running 1 month (one `REST_OPTION`)     
readonly RESUBMIT="0"          # Do not re-submit
readonly DO_SHORT_TERM_ARCHIVING=false

For a long production run, this section might look like:

...

• readonly PELAYOUT="L": 1=single processor, S=small, M=medium, L=large, X1=very large, X2=very very large. Use S for short tests. Full simulations should typically use M or L. The size determines how many nodes will be used. The exact number of nodes will differ amongst machines.

• readonly WALLTIME="28:00:00" : maximum wall clock time requested for the batch jobs.

• readonly PROJECT="e3sm" : accounting project of the batch jobs.

# Case name

• readonly CASE_NAME=${CHECKOUT}.${DESCRIPTOR}.${MACHINE}: sets the case name. This will look like20210205.v2_test02.piControl.ne30pg2_EC30to60E2r2.chrysalis using the example values on this page.

  • If you are not comparing the same case on different machines, you can exclude the .${MACHINE} part.

...

# Sub-directories (leave unchanged)

This section should typically not be changed:

readonly CASE_SCRIPTS_DIR=${CASE_ROOT}/case_scripts # Where files for your particular simulation will go.
readonly CASE_BUILD_DIR=${CASE_ROOT}/build          # All the stuff to compile. The executable will be there.
readonly CASE_RUN_DIR=${CASE_ROOT}/run              # Where all the output will be. Most components will have their own log files.
readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive      # Where archives will go.

...

• build: all the stuff to compile. The executable is also there (e3sm.exe).

• case_scripts: the files for your particular simulation.

• run: where all the output will be. Most components (atmosphere, ocean, etc.) have their own log files. The coupler exchanges information between the components. The top level log file will be of the form run/e3sm.log.*. Log prefixes correspond to components of the model:

  • atm: atmosphere

  • cpl: coupler

  • ice: sea ice

  • lnd: land

  • ocn: ocean

  • rof: river runoff

...