Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

EAM and SCREAM uses the HOMME dycore package which contains the EAM v1 hydrostatic dycore (PREQX) and the proposed v2 nonhydrostatic dycore (THETA).  These dycores share much of the HOMME infrastructure and have several timesteps:  

...

  1. Run 10 days with very small, inefficient timesteps.   Take dt_remap=dt_dyn and use small values for  dt_dyn, dt_vis and dt_vis_tom.  Create a new IC file (see INITHIST in EAM).   Use this new IC file for all future runs below
  2. dt_vis_tom:  Use the CFL condition (take S=2) printed in atm.log to determine the largest safe value of dt_vis_tom.   In practice the code appears to be unstable around S=3.3, and it is good not to run right at the stability limit.     
  3. dt_dyn:   Keeping all other timesteps below their known stable limits (which takes some care due to how they are all set) make a serious of runs increasing just dt_dyn until the run crashes.  May need to make relatively long runs (several months) to ensure stability.  
  4. dt_vis:  Keeping all other timesteps fixed  find the largest stable value.  For THETA, with the recommended viscosity coefficients, dt_vis = dt should be stable.  For some RRM grids, dt_vis might be need to be slightly smaller due to mesh distortion.  When the viscous CFL is violated (dt_vis too large), the run usually crashes within a couple of steps.  
  5. The procedure outlined above can find timesteps that are borderline unstable, but don’t blow up do to various dissipation mechanisms.  Hence it is a good idea to run 3 months, and look at the monthly mean OMEGA500 from the 3rd month. This field will be noisy, but there should not be any obvious grid artifacts.  Weak instabilities can be masked by the large transients in flow snapshots, so it best to look at time averages.  
  6. dt_remap:  Using 2*dt_dyn is a conservative choise that is well tested up to ne240.   Decreasing dt_remap results in more frequent vertical remaps resulting in increased vertical dissipation.   If dt_remap is too large, the code may crash with negative layer thickness errors.  The code has a "dp3d" limiter that can prevent some of these crashes.  If this limiter is triggered (warnings in e3sm.log file), that 's a sign that can mean either dt_remap is too large of one of the CFL conditions has been violated and the code is unstable       

During this tuning process, it is useful to compare the smallest ‘dx’ from the atmosphere log file to the smallest ‘dx’ from the global uniform high resolution run.  Use the ‘dx’ based on the singular values of Dinv, not the ‘dx’ based on element area. If the ‘dx’ for newmesh.g is 20% smaller than the value from the global uniform grid, it suggests the advective timesteps might need to be 20% smaller.  The code prints out CFL estimates that are rough approximation that can be used to check if you are in the correct ballpark.

...

With RRM grids, the timesteps will be controlled by the highest resolution region.  So with an RRM grid with refinement down to NE120, the timesteps should be close to what we run on a uniform cubed-sphere NE120 grid.   The timesteps may need to be slightly smaller because of the deformed elements in the transition region.   With a hiqh quality RRM mesh ( Max Dinv-based element distortion metric <= 4, see Generate the Grid Mesh (Exodus) File for a new Regionally-Refined Grid) we can usually run with the expected dt_dyn and dt_tracer values, and only the viscosity timesteps need to be slightly reduced.  




Recommended settings (THETA)   

...

ResolutionTimestepsNamelist settingsNotesTested?
1 degree (NE30)

dtime=1800  (ATM_NCPL=72)

dt_remap=600
dt_tracer=1800
dt_dyn=dt_vis=dt_vis_q=300
dt_vis_tom=300



nu=1e15
nu_top = 2.5e5    

se_tstep=300





With dt_remap=1800, we see occasional (every 2-3 years) dp3d limiter activation, meaning that the model levels are approaching zero.  This appears to be due to strong divergence above tropical cyclones created by one of the parameterizations.

HS+topo(72L):  H and NH 

F-EAMv1-AQP1:  H and NH 

FAV1C-L:  H and NH


1/4 degree (NE120)

dtime=900  (ATM_NCPL=144)

dt_remap=150
dt_tracer=450..650
dt_dyn=dt_vis=dt_vis_q=75
dt_vis_tom=75

nu=1.5e13
nu_top=1e5

se_tstep=75



CFL estimates suggest:

dt_vis_tom*nu_top <= 31*2.5e5

nu_top=2.5e5 would need
hypervis_subcycle_tom=3


HS+topo(72L):  H and NH (with dt_remap=75 and theta limiter to handle unphysical boundary layer)

F-EAMv1-AQP1:  H  and NH, both 72 and 128 levels (1+ years)

FC5AV1C-H01B:  NH 72L runs several years

12km

(NE256)

dtime=480?
ATM_NCPL=180

dt_remap=80
dt_tracer=240..360
dt_dyn=dt_vis=dt_vis_q=40
dt_vis_tom = 40

nu=1.6e12
nu_top=4e4
se_tstep=40





nu_tom=4e4 is running at the code's estimate of the CFL limit with S=1.9

dt=dt_dyn/ dt_remap/ dt_tracer / dt_phys

F-EAMv1-AQP1:
H-128L running with default timesteps. (1 month)  

NH-128L:

  • tstep_type=9:  Runs with dt=37.5/75/300/600.  crashes with dt=40/40/240/480 after 6d, dp3d<0 around layer 30.   
  • with tstep_type=10: dt=40/40/240/480  ran for 1 month.  


6km (NE512)dt_remap=40
dt_tracer=120..180
dt_dyn=dt_vis=dt_vis_q=20
dt_vis_tom=20

nu=2e11
nu_top=2e4
se_tstep=20



CFL estimates suggest:

dt_vis_tom*nu_top <= 1.7*2.5e5,

nu_top=2.5e5 needs hypervis_subcycle_tom=13 

F-EAMv1-AQP1:

H-128L:  dt=20/40/120/240 good.  

NH-128L: 

  • tstep_type=9:  dt=20/40/120/240: crash bad EOS 2.4days.  dt=18.75/37.5/150/300, ran 20days.
  • tstep_type=10: dt=20/40/120/240 crash 3.1days.  bad dp3d layer 75
3km (NE1024)

dt_remap=20
dt_tracer=60..90
dt_dyn=dt_vis=dt_vis_q=10
dt_vis_tom=10

nu=2.5e10
nu_top=1e4
se_tstep=10


CFL estimates suggest:

dt_vis_tom*nu_top <= 0.43*2.5e5

with nu_top=2.5e5, 
hypervis_subcycle_tom=24



...