EAM and SCREAM uses the HOMME dycore package. HOMME has both Fortran and C++ versions with different options for the prognostic thermodynamic variable (preqx use temperature while theta-l uses potential temperature) and those in turn have some additional capabilities.
...
dt_vis, dt_vis_q, dt_vis_tom: These timesteps have a CFL condition that depends on the horizontal resolution and the viscosity coefficient. In the case of the TOM diffusion, the viscosity coefficient depends on the number of levels, bringing in an additional dependency.
- With the default HV coefficient of 1e15 m^4/s for nu, nu_div, nu_q, nu_p and nu_s at NE30, and assuming this coefficient is scaled as dx^3 or dx^3.2 power under mesh refinement:recommended hyper viscosity settings, one should take:
- dt_vis = dt_vis_q = dt_dyn
- The code prints an estimate of the viscous CFL condition in the log file, in the form "dt < S * max_eigen_value", where S depends on the timestepping method. Hyperviscosity is applied with forward Euler, suggesting S=2. However, numerical experiments on cubed sphere grids suggest the HV operator goes unstable at S=.80 and so we recommend taking S<.70 when estimating the maximum stable timestep. For RRM grids S might need to be smaller.
- The TOM sponge layer uses a laplacian operator with coefficient, nu_top. We used to keep this coefficient fixed (nu_top=2.5e5) for all resolutions. But above 1/4 degree, this creates a severe CFL condition. For v2 and beyond, we now recommend running 2.5e5 for resolutions up to 1 degree (NE30) and for higher resolution run with a CFL number of 1.9. This number is based on the CFL condition printed by the code. Experiments suggest the operator is stable up to S=2.89, but unstable at S=3.38. NOTE regarding RRM grids: The HV coefficient is strongly resolution dependent, decreasing like dx^3 or dx^3.2. For RRM grids we thus use a tensor hyperviscosity formulation that contains this scaling and applies it locally as the grid resolution changes. Tensor hyperviscosity is enabled by setting hypervis_scaling=3. For the tensor hyperviscosity, the coefficients (nu, nu_div, nu_q, nu_p, nu_s) are now specified in 1/s units that are independent of resolution. .89, but unstable at S=3.38.
- In V1, we had nu_div >> nu, requiring a much smaller dt_vis that must be tuned specially. This is not needed in V2.
Recommended timestep tuning for new grids
...
With RRM grids, the timesteps will be controlled by the highest resolution region. So with an RRM grid with refinement down to NE120, the timesteps should be close to what we run on a uniform cubed-sphere NE120 grid. The timesteps may need to be slightly smaller because of the deformed elements in the transition region. With a hiqh quality RRM mesh ( Max Dinv-based element distortion metric <= 4, see Generate the Grid Mesh (Exodus) File for a new Regionally-Refined Grid) we can usually run with the expected dt_dyn and dt_tracer values, and only the viscosity timesteps need to be slightly reduced.
Spreadsheet for looking at scaling with resolution of constant and tensor coefficient HV:
https://docs.google.com/spreadsheets/d/1LHTl2_A065pfdWC69OHmXvNL_v1484cXg7ZowhyEbPU/edit?usp=sharing
...
| Resolution | Timesteps | Namelist settings | Notes | Tested? |
|---|---|---|---|---|
| 1 degree (NE30) | dtime=1800 (ATM_NCPL=48) dt_tracer=1800 | nu_top = 2.5e5 se_tstep=300 dt_tracer_factor = 6 hypervis_scaling=0 | E3SMv2 NE30 does not use tensorHV to avoid having to retune. Should change in v3. With dt_remap=1800, we see occasional (every 2-3 years) dp3d limiter activation, meaning that the model levels are approaching zero. This appears to be due to strong divergence above tropical cyclones created by one of the parameterizations. | HS+topo(72L): H and NH (H can run at t 360s but not 400s with either tstep_type=4 or 5). dt_remap=600 runs with no limiter warnings, while dt_remap=900 crashes with dp3d<0 at surface. F-EAMv1-AQP1: H and NH FAV1C-L: H and NH |
| NE45 | dtime=1200. (ATM_NCPL=72) dt_tracer=1200 dt_vis_tom=200 | nu_top=2.5e5 se_tstep=200 | ||
| 1/4 degree (NE120) | dtime=900 (ATM_NCPL=96) dt_remap=150 | nu_top=1e5 se_tstep=75 dt_tracer_factor = 6 | CFL estimates suggest: dt_vis_tom*nu_top <= 31*2.5e5 nu_top=2.5e5 would need | HS+topo(72L): H and NH (with dt_remap=75 and theta limiter to handle unphysical boundary layer) F-EAMv1-AQP1: H and NH, both 72 and 128 levels (1+ years) FC5AV1C-H01B: NH 72L runs several years. |
12km (NE256) | dtime=600 (ATM_NCPL=144) dt_tracer=200 NOTE: these defaults were updated 2021/9 based on SCREAM v0.1 3km runs. But NE256 is known to run stably at slightly larger timsteps: dt_tracer=300 | nu_top=4e4 dt_tracer_factor=6 | nu_tom=4e4 is running at the code's estimate of the CFL limit with S=1.9 | F-EAMv1-AQP1:
FC5AV1C-H01B: NH 128L run for several months dt=37.5/75/300/600. Occasional problems near coastlines - considering ( ) reducing dtime, increasing HV, tunning CLUBB |
| 6km (NE512) | dtime=300 (ATM_NCPL=288) dt_tracer=100 | nu_top=2e4 dt_tracer_factor = 6 | CFL estimates suggest: dt_vis_tom*nu_top <= 1.7*2.5e5, nu_top=2.5e5 needs hypervis_subcycle_tom=13 | F-EAMv1-AQP1:
FC5AV1C-H01B: NH 128L run for 1 day with dt=18.75/37.5/150/300, then NaNs in microphysics (not yet debugged) |
| 3km (NE1024) | dtime=100 (ATM_NCPL=576) dt_tracer=50 | nu_top=1e4 dt_tracer_factor = 6 | CFL estimates suggest: dt_vis_tom*nu_top <= 0.43*2.5e5 with nu_top=2.5e5, | F-EAMv1-AQP1:
FC5AV1C-H01B: SCREAM v0: run for 40 days with constantHV, dt=9.375/18.75/75/75. SCREAM v0.1: switch to tensorHV (slightly less diffusion), needs dt_dyn<=9s, dt_tracer<60 |
| RRM | dtime=? dycore timesteps should be set based on the finest region in the RRM. | nu_top=Uncertain - needs more research. Should probably switch to tensor laplacian. For NE30→NE120 grids, start with NE120 constant coefficient value, 1e5. | RRM uses a tensor HV formulation which scales with resolution dx^3.0 (For preqx, we used a dx^3.2 scaling. ) To determine the effective HV coefficient at a given resolution "dx", use: nu_tensor = nu_const *( 2*rearth /((np-1)*dx))^{hv_scaling} * rearth^{-4.0}. i.e. tensor nu=3.4e-8 when used at 1 degree resolution (dx=110,000m, np=4, rearth=6.376e6) is equivalent to 1e15 m^4/s. |
...