This page documents how to run SCREAMv1 on supported machines.
Notes:
cori optimized build has an internal compiler error (in shoc_assumed_pdf.cpp?). Avoid this by building in debug mode.
perlmutter optimized build yields corrupted answers (really hot planet). Avoid this by building in debug mode.
ne120 fails with OOM(?) errors right now when SPA is active. Delete that proc in namelist_scream.xml.
Step 1: Conda environment
The SCREAMv1 build requires a handful of python libraries. Creating a conda environment with those packages installed is the easiest way to create an appropriate environment. For example,
conda create -n scream_v1_build pyyaml pylint psutil
will create an environment named scream_v1_build
with the appropriate packages. Then activate the environment before building.
conda activate scream_v1_build
Note that SCREAM also needs a ton of info about compilers, netcdf, etc but that info is acquired automatically on supported machines.
Step 2: Define convenience variables
In order to provide commands below which should work for everyone on all supported machines, we create some user-specific variables below. Change these as needed:
export CODE_ROOT=~/gitwork/scream/ #or wherever you cloned the scream repo export COMPSET=F2010-SCREAMv1 #or whatever compset you want export RES=ne4_ne4 #or whatever resolution you want export CASE_NAME=${RES}.${COMPSET}.test1 #name for your simulation. export PECOUNT=96x1 # Number of MPIs by number of threads. Should be divisible by node size
Resolution options:
Resolution | Grid name (aka $RES) | |
---|---|---|
ne4 | ne4_ne4 | |
ne30 | ne30_ne30 | |
ne120 | ne120_r0125_oRRS18to6v3 | |
ne256 | ||
ne512 | ||
ne1024 |
Suggested PECOUNTs (not necessarily performant, just something to get started)
ne4 (max = 96) | ne30 (max = 5,400) | ne120 (max = 86,400) | ne256 (max = 393,216) | ne512 (max = 1,572,864) | ne1024 (max = 6,291,456) | |
---|---|---|---|---|---|---|
cori-knl (68 cores/node; 96+16 GB/node) | 16x1 | |||||
perlmutter (64 cores/node; 4 GPUs/node; 256 GB/node) | ||||||
syrah (16 cores/node; 64 GB/node) | 32x1 | 160x1 | 320x1 | |||
quartz (36 cores/node; 128 GB/node) | 72x1 | 180x1 | 360x1 | |||
summit (8 cores/node?; 6 GPUs/node; 512+96 GB/node) |
Step 3. Create the Case
From the location you want to build+run the model, issue:
${CODE_ROOT}/cime/scripts/create_newcase --case ${CASE_NAME} --compset ${COMPSET} --res ${RES} --pecount ${PECOUNT} --walltime 00:30:00 --queue pdebug
Then cd ${CASE_NAME}
Step 4: Change CIME settings (if Desired)
You shouldn’t need to change anything to run, but some things you may want to change are:
./xmlchange ATM_NCPL=288 ./xmlchange DEBUG=TRUE #debug rather than optimized build. ./xmlchange JOB_QUEUE=pdebug ./xmlchange JOB_WALLCLOCK_TIME=0:30:00 ./xmlchange STOP_OPTION=ndays #how long to run for ./xmlchange STOP_N=1 ./xmlchange HIST_OPTION=ndays #how often to write cpl.hi files ./xmlchange HIST_N=1
The point of these changes are (respectively):
change the atm timestep to 288 steps per day (300 sec). This needs to be done via ATM_NCPL or else the land model will get confused about how frequently it is coupling with land
compile in debug mode. Will run 10x slower but will provide better error messages. And doesn’t run in any other mode on some machines.
change the default queue and wallclock from the standard queue with 1 hr walltime to debug queue and its max of 30 min walltime to get through the queue faster.
change the default length of the run from just a few steps to 1 day (or whatever you choose). This change is made in both env_run.xml and env_test.xml because case.submit seems to grab for one or the other file according to confusing rules - easier to just change both.
HIST_OPTION and HIST_N set the frequency of coupler snapshots (cpl.hi) files, which are useful for figuring out whether SCREAM is getting or giving bad data from/to the surface models
Step 4: Change SCREAM settings
As of , this is done by modifying namelist_scream.xml either by hand or by using the atm-config-chg
function which now comes bundled when you create a case. Explore namelist_scream.xml for variables you might want to change (but you shouldn’t have to change anything to run).
Step 6: Config/Compile/Run
Now setup, build, and run. You can setup and build before most or all of the above customization.
./case.setup ./case.build ./case.submit
You can check it’s progress via squeue -u <username>
on many systems. Model output will be in the run subdirectory.