Page Comparison

This page documents how to run SCREAMv1 on supported machines.

Notes:

...

ne512 and ne1024 are too big to use SPA right now (will OOM). Run them with “no SPA” (prescribed drop #, no aerosol in radiation) by using the F2010-SCREAMv1-noAero compset. Several fixes are in the works.

1. For perlmutter: you need to use the gnugpu compiler and set --gpu-bind=none in env_batch.xml:

   Code Block
   <directives compiler="gnugpu"> <directive> --gpus-per-task=1</directive> <directive> --gpu-bind=none</directive>

Step 1: Environment:

The SCREAMv1 build requires a handful of python libraries. It also requires a ton of other dependencies, but CIME handles these automatically. Here’s what you need to do on each machine:

...

The point of these changes are (respectively):

1. change the atm timestep to 288 steps per day (300 sec). This needs to be done via ATM_NCPL or else the land model will get confused about how frequently it is coupling with land

2. compile in debug mode. Will run 10x slower but will provide better error messages. And doesn’t run in any other mode on some machines.

3. change the default queue and wallclock from the standard queue with 1 hr walltime to debug queue and its max of 30 min walltime to get through the queue faster. Note that the format for summit wallclock limits is hh:mm instead of hh:mm:ss on other machines.

4. change the default length of the run from just a few steps to 1 day (or whatever you choose). This change is made in both env_run.xml and env_test.xml because case.submit seems to grab for one or the other file according to confusing rules - easier to just change both.

5. HIST_OPTION and HIST_N set the frequency of coupler snapshots (cpl.hi) files, which are useful for figuring out whether SCREAM is getting or giving bad data from/to the surface models

6. NTASKS is the number of MPI tasks to use (which sets the number of nodes submit asks for). You can also set this via --pecount setting in create_newcase.

7. Changing the PIO_NETCDF_FORMAT to 64bit_data is needed at very high resolutions to avoid exceeding max variables size limits.

Step 5: Configure the case

...

Make the following ./xmlchange commands:

1. ./xmlchange ATM_NCPL . Note that this is the number of atm timesteps per day so the model timestep (commonly known as dtime) is 86400 sec/ATM_NCPL. See the dycore settings link at the end of this section for guidance on choosing dtime.

In namelist_scream.xml, make the following changes:

1. Change Vertical__Coordinate__Filename to use the initial condition file for your new resolution

2. Change Filename under Initial__Conditions → Physics__GLL subsection to also use that new initial condition file

3. Change SPA__Remap__File to use one appropriate to map ne30 to your new resolution

4. Change se_ne as appropriate

5. change se_tstep and nu_top (recommended defaults for these and dtime are given in the awesome table on the EAM's HOMME Dycore Recommended Settings (THETA) page

** v1 ne256 notes (06/09/2022) **

Currently, getting ne256 v1 through initialization requires:

1. Setting:

   Code Block
   ./xmlchange EPS_AGRID=1.e-9

2. Setting:

   Code Block
   ./atmchange cubed_sphere_map=2

3. For SPA remap file point to (has been uploaded to E3SM input data server as of 06/09/2022) atm/scream/init/map_ne30np4_to_ne256np4_mono.20220609.nc

4. Make sure you are compiling with 128 vertical levels (currently, SCREAM compiles with 72 layers by default) .

   Code Block
   ./xmlchange SCREAM_CMAKE_OPTIONS="SCREAM_NP 4 SCREAM_NUM_VERTICAL_LEV 128 SCREAM_NUM_TRACERS 10"