EAM and SCREAM uses the HOMME dycore package which contains the EAM v1 hydrostatic dycore (PREQX) and the proposed v2 nonhydrostatic dycore (THETA). These dycores share much of the HOMME infrastructure and have several timesteps:
- dtime: The physics timestep. set in env_run.XML via ATM_NCPL. Note: dtime can be entered in user_nl_cam, but it will be silently ignored.
- dt_remap: The Lagrangian vertical level remap timestep
- dt_tracer: The tracer advection timestep
- dt_dyn Dynamics (u,v,temperature,mass) timestep
- dt_vis Hyperviscosity timestep for dynamics
- dt_vis_tom top-of-model Laplacian diffusion (sponge layer)
- dt_vis_q hyperviscosity timestep for tracers
These timesteps are controlled by various splitting parameters. In EAM, "dtime" is set outside the dycore and used to define all other timesteps. The splitting parameters enforce the fact that the various timesteps must evenly divide other timesteps.
See also Timestepping within the ACME v1 Atmosphere Model
PREQX dycore: (CAM_TARGET=preqx)
User namelist inputs: nsplit, rsplit, qsplit, hypervis_subcycle, hypervis_subcycle_q
Resulting timesteps:
- dt_remap = dtime/nsplit
- dt_tracer = dt_remap/rsplit
- dt_dyn = dt_tracer/qsplit
- dt_vis = dt_dyn/hypervis_subcycle
- dt_vis_tom = dt_vis
- dt_vis_q = dt_tracer/hypervis_subcycle_q
THETA dycore: (CAM_TARGET=theta-l, changing to theta)
In EAM v2, we will be transitioning to the "theta" dycore, which supports both hydrostatic and nonhydrostatic dynamics and semi-Lagrangian tracer transport. The theta dycore also supports the ability to have dt_tracer > dt_remap, and introduces a separate timestep for the TOM sponge layer.
Enabling the THETA dycore in E3SM:
To enable the THETA dynamical core:
- ./xmlchange CAM_TARGET=theta-l
- for SCREAM physics or high resolution: semi_lagrange_cdr_alg=21
Default settings will use SL transport and Hydrostatic mode. To switch to Eulerian transport add to user_nl_cam
- transport_alg=0, hypervis_subcycle_q=1, dt_tracer_factor=1
To switch to NH mode:
- theta_hydrostatic_mode=.false., tstep_type=9.
In NH mode, tstep_type=9 is recommended. Other options:
- tstep_type=5 is the 3rd order accurate KGU53 explicit method used for hydrostatic simulations. It can be used with NH simulations with very small timestepes, ~0.5s
- tstep_type=9 is an IMEX method, combining an explicit Butcher table (KGU53, used for most equations) and an implicit Butcher table (used for terms responsible for vertically propagating acoustic waves not present in hydrostatic models).
- tstep_type=10 is another IMEX method which uses KG52 for the explicit table. It has a slightly larger stability region.
Detailed description of timesteps and viscosity parameters
The PREQX and tHETA dycores share much of the HOMME infrastructure and have several timesteps:
- dtime: The physics timestep. set in env_run.XML via ATM_NCPL. Note: dtime can be entered in user_nl_cam, but it will be silently ignored.
- dt_remap: The Lagrangian vertical level remap timestep
- dt_tracer: The tracer advection timestep
- dt_dyn Dynamics (u,v,temperature,mass) timestep
- dt_vis Hyperviscosity timestep for dynamics
- dt_vis_tom top-of-model Laplacian diffusion (sponge layer)
- dt_vis_q hyperviscosity timestep for tracers
These timesteps are controlled by various splitting parameters. In EAM, "dtime" is set outside the dycore and used to define all other timesteps. The splitting parameters enforce the fact that the various timesteps must evenly divide other timesteps.
See also Timestepping within the ACME v1 Atmosphere Model
PREQX dycore: (CAM_TARGET=preqx)
User namelist inputs: nsplit, rsplit, qsplit, hypervis_subcycle, hypervis_subcycle_q, hypervis_subcycle_tom
Resulting timesteps:
- dt_remap = dtime/nsplit
- dt_tracer = dt_remap/rsplit
- dt_dyn = dt_tracer/qsplit
- dt_vis = dt_dyn/hypervis_subcycle
- dt_vis_tom = dt_vis
- dt_vis_q = dt_tracer/hypervis_subcycle_q
- dt_vis_tom = dt_dyn / max(hypervis_subcycle, hypervis_subcycle_tom)
...
THETA dycore: (CAM_TARGET=theta-l, changing to theta)
In EAM v2, we will be transitioning to the "theta" dycore, which supports both hydrostatic and nonhydrostatic dynamics and semi-Lagrangian tracer transport. The theta dycore also supports the ability to have dt_tracer > dt_remap, and introduces a separate timestep for the TOM sponge layer.
User namelist inputs: tstep, dt_remap_factor, dt_tracer_factornsplit, rsplit, qsplit, hypervis_subcycle, hypervis_subcycle_q, hypervis_subcycle_tom
Resulting timesteps:
- dt_remap = tstep*= dtime/nsplit
- dt_tracer = dt_remap/rsplit
- dt_factor dyn ( must evenly divide EAM's dtime, checked at runtime)
- dt_tracer = tstep*dt_tracer_factor ( must evenly divide EAM's dtime, checked at runtime)
- dt_dyn = tstep
- dt_vis = dt_= dt_tracer/qsplit
- dt_vis = dt_dyn/hypervis_subcycle
- dt_vis_q = dt_tracer/hypervis_subcycle_q
- dt_vis_tom = dt_dyn / max(hypervis_subcycle, hypervis_subcycle_tom)
An additional constraint is that either dt_remap must evenly divide dt_tracer or dt_tracer must evenly divide dt_remap.
CFL Conditions
dt_dyn: The CFL condition for dt_dyn with uniform resolutions is well understood New namelist variables: EAM now allows the user to directly set the main dycore timesteps with new namelist options "tstep, dt_remap_factor, dt_tracer_factor" replacing "nsplit, rsplit and qsplit":
User namelist inputs: tstep, dt_remap_factor, dt_tracer_factor, hypervis_subcycle, hypervis_subcycle_q, hypervis_subcycle_tom
Resulting timesteps:
- dt_remap = tstep*dt_remap_factor ( must evenly divide EAM's dtime, checked at runtime)
- dt_tracer = tstep*dt_tracer_factor ( must evenly divide EAM's dtime, checked at runtime)
- dt_dyn = tstep
- dt_vis = dt_dyn/hypervis_subcycle
- dt_vis_q = dt_tracer/hypervis_subcycle_q
- dt_vis_tom = dt_dyn / max(hypervis_subcycle, hypervis_subcycle_tom)
An additional constraint is that either dt_remap must evenly divide dt_tracer or dt_tracer must evenly divide dt_remap.
CFL Conditions
dt_dyn: The CFL condition for dt_dyn with uniform resolutions is well understood and depends on the horizontal resolution and the speed of the Lamb wave (~340m/s). This assumes the use of the KG5 RK method (tstep_type=5) for hydrostatic, and the related IMEX method (tstep_type=9 or 10) for NH.
...
- With the default HV coefficient of 1e15 m^4/s for nu, nu_div, nu_q, nu_p and nu_s at NE30, and assuming this coefficient is scaled as dx^3 or dx^3.2 power under mesh refinement:
- dt_vis = dt_vis_q = dt_dyn
- In the PREQX dycore, we usually run with nu_div >> nu, requiring a much smaller dt_vis that must be tuned specially. This should not be needed with the THETA dycore
- The TOM sponge layer uses a laplacian operator with a fixed (resolution independent) coefficient, nu_top. This creates a severe CFL condition - if we double the resolution, we must decrease dt_vis_tom by 4x. For a 72L configuration, hypervis_subcycle_tom (THETA) or hypervis_subcycle (PREQX) must be set so that:
- NE30: dt_vis_tom <= 500
- NE60: dt_vis_tom <= 125
- NE120: dt_vis_tom <= 31
- NE256: dt_vis_tom <= 6.9
- NE512: dt_vis_tom <= 1.7
- NE1024: dt_vis_tom <= 0.43
- Using the nu_top CFL condition printed by the code, experiments suggest the operator is stable up to S=2.89, but unstable at S=3.38.
- For the theta model we recommend (dependent on future testing) to take nu_top=2.5e5 up to NE30. For resolutions > NE30, choose nu_top at the CFL limit using S=1.9
- NOTE regarding RRM grids: The HV coefficient is strongly resolution dependent, decreasing like dx^3 or dx^3.2. For RRM grids we thus use a tensor hyperviscosity formulation that contains this scaling and applies it locally as the grid resolution changes. Tensor hyperviscosity is enabled by setting hypervis_scaling>0 (typically 3 or 3.2). For the tensor hyperviscosity, the coefficients (nu, nu_div, nu_q, nu_p, nu_s) are now specified in 1/s units that are indepdent of resolution. For hypervis_scaling=3.2, we currently recommend setting all coefficients to 8e-8. Currently E3SM only uses tensor hyperviscosity for RRM grids, but we should consider using it for all grids.
Recommended timestep tuning for new grids
Do not get mislead by spinup issues! Determining stable timestep values is complicated by spinup. If the initial condition file is very much out of balance (say it came from intepolation from observations using different topography) then you may require very small timesteps and larger hyperviscosity coefficients in order to get past an initial adjustment period. Only perform the tuning for the final timesteps with a spun up initial condition file.
...
During this tuning process, it is useful to compare the smallest ‘dx’ from the atmosphere log file to the smallest ‘dx’ from the global uniform high resolution run. Use the ‘dx’ based on the singular values of Dinv, not the ‘dx’ based on element area. If the ‘dx’ for newmesh.g is 20% smaller than the value from the global uniform grid, it suggests the advective timesteps might need to be 20% smaller. The code prints out CFL estimates that are rough approximation that can be used to check if you are in the correct ballpark.
RRM Grids
With RRM grids, the timesteps will be controlled by the highest resolution region. So with an RRM grid with refinement down to NE120, the timesteps should be close to what we run on a uniform cubed-sphere NE120 grid. The timesteps may need to be slightly smaller because of the deformed elements in the transition region. With a hiqh quality RRM mesh ( Max Dinv-based element distortion metric <= 4, see Generate the Grid Mesh (Exodus) File for a new Regionally-Refined Grid) we can usually run with the expected dt_dyn and dt_tracer values, and only the viscosity timesteps need to be slightly reduced.
Recommended settings (THETA)
All resolutions: (default for theta-l dycore in master as of https://github.com/E3SM-Project/E3SM/pull/3368 )
se_ftype=2
se_limiter_option=9
cubed_sphere_map=2 (change to 0 in user_nl_cam when using old compsets with ESMF maps)
transport_alg=12
semi_lagrange_cdr_alg=3 (2020/4: default changed to 3 reproduce some aspects of the v1 climate in v2)
semi_lagrange_cdr_alg=21 (recommended for SCREAM)
semi_lagrange_nearest_point_lev = 256
semi_lagrange_hv_q = 1
theta_advect_form =1
vert_remap_q_alg=10
nu_div=nu_p=nu_q=nu_s=-1 (use nu for all values)
nu, nu_top: Resolution dependent
se_nstep=-1 ( timesteps set via se_tstep, dt_remap_factor, dt_tracer_factor)
qsplit=-1
rsplit=-1
se_tstep (Resolution dependent)
dt_tracer_factor (Resolution dependent)
dt_remap_factor = 2
...
To enable the THETA dynamical core:
- ./xmlchange CAM_TARGET=theta-l
- for SCREAM physics or high resolution: semi_lagrange_cdr_alg=21
Default settings will use SL transport and Hydrostatic mode. To switch to Eulerian transport add to user_nl_cam
- transport_alg=0, hypervis_subcycle_q=1, dt_tracer_factor=1
To switch to NH mode:
...
.com/E3SM-Project/E3SM/pull/3368 )
se_ftype=2
se_limiter_option=9
cubed_sphere_map=2 (change to 0 in user_nl_cam when using old compsets with ESMF maps)
transport_alg=12
semi_lagrange_cdr_alg=3 (2020/4: default changed to 3 reproduce some aspects of the v1 climate in v2)
semi_lagrange_cdr_alg=21 (recommended for SCREAM)
semi_lagrange_nearest_point_lev = 256
semi_lagrange_hv_q = 1
theta_advect_form =1
vert_remap_q_alg=10
nu_div=nu_p=nu_q=nu_s=-1 (use nu for all values)
nu, nu_top: Resolution dependent
se_nstep=-1 ( timesteps set via se_tstep, dt_remap_factor, dt_tracer_factor)
qsplit=-1
rsplit=-1
se_tstep (Resolution dependent)
dt_tracer_factor (Resolution dependent)
dt_remap_factor = 2
hypervis_subcycle_q = dt_tracer_factor
hypervis_subcycle=1
hypervis_subcycle_tom=1
Resolution | Timesteps | Namelist settings | Notes | Tested? |
---|---|---|---|---|
1 degree (NE30) | dtime=1800 (ATM_NCPL=48) dt_tracer=1800 | nu=1e15 se_tstep=300 dt_tracer_factor = 6 | With dt_remap=1800, we see occasional (every 2-3 years) dp3d limiter activation, meaning that the model levels are approaching zero. This appears to be due to strong divergence above tropical cyclones created by one of the parameterizations. | HS+topo(72L): H and NH F-EAMv1-AQP1: H and NH FAV1C-L: H and NH |
1/4 degree (NE120) | dtime=900 (ATM_NCPL=96) dt_remap=150 | nu=1.5e13 se_tstep=75 dt_tracer_factor = 6 | CFL estimates suggest: dt_vis_tom*nu_top <= 31*2.5e5 nu_top=2.5e5 would need | HS+topo(72L): H and NH (with dt_remap=75 and theta limiter to handle unphysical boundary layer) F-EAMv1-AQP1: H and NH, both 72 and 128 levels (1+ years) FC5AV1C-H01B: NH 72L runs several years. |
12km (NE256) | dtime=600 (ATM_NCPL=144) dt_tracer=300 | nu=1.6e12 dt_tracer_factor=8 | nu_tom=4e4 is running at the code's estimate of the CFL limit with S=1.9 | F-EAMv1-AQP1:
FC5AV1C-H01B: NH 128L run for several months dt=37.5/75/300/600. Occasional problems near coastlines - considering ( ) reducing dtime, increasing HV, tunning CLUBB |
6km (NE512) | dtime=300 (ATM_NCPL=288) dt_tracer=150 | nu=2e11 dt_tracer_factor=8 | CFL estimates suggest: dt_vis_tom*nu_top <= 1.7*2.5e5, nu_top=2.5e5 needs hypervis_subcycle_tom=13 | F-EAMv1-AQP1:
FC5AV1C-H01B: NH 128L run for 1 day with dt=18.75/37.5/150/300, then NaNs in microphysics (not yet debugged) |
3km (NE1024) | dtime=150 (ATM_NCPL=576) dt_tracer=75 | nu=2.5e10 dt_tracer_factor=8 | CFL estimates suggest: dt_vis_tom*nu_top <= 0.43*2.5e5 with nu_top=2.5e5, | F-EAMv1-AQP1:
FC5AV1C-H01B: SCREAM physics: run for several days with dt=9.375/18.75/75/75. Frequent (daily) problems near coastlines - considering ( ) different topography, reducing dtime, increasing HV |
RRM | dtime=? dycore timesteps should be set based on the finest region in the RRM. | hyperviscosity_scaling=3.0 nu=3.4e-8 nu_top=Uncertain - needs more research. Should probably switch to tensor laplacian. For NE30→NE120 grids, start with NE120 constant coefficient value, 1e5. | RRM uses a tensor HV formulation which scales with resolution dx^3.0 (For preqx, we used a dx^3.2 scaling. ) To determine the effective HV coefficient at a given resolution "dx", use: nu_tensor = nu_const *( 2*rearth /((np-1)*dx))^{hv_scaling} * rearth^{-4.0}. i.e. tensor nu=3.4e-8 when used at 1 degree resolution (dx=110,000m, np=4, rearth=6.376e6) is equivalent to 1e15 m^4/s. |
PREQX Default Settings (for reference)
se_ftype=2
se_limiter_option=8
cubed_sphere_map=0 (2 when using TR maps or RRM Grids)
...