Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

EAM and SCREAM uses the HOMME dycore package which contains the EAM v1 hydrostatic dycore (PREQX) and the proposed v2 nonhydrostatic dycore (THETA).  These dycores share much of the HOMME infrastructure and have several timesteps:  

...

  • With the default HV coefficient of 1e15 m^4/s for nu, nu_div, nu_q, nu_p and nu_s at NE30, and assuming this coefficient is scaled as dx^3 or dx^3.2 power under mesh refinement:
    • dt_vis = dt_vis_q = dt_dyn      
  • In the PREQX dycore, we usually run with nu_div >> nu, requiring a much smaller dt_vis that must be tuned specially.  This should not be needed with the THETA dycore
  • The TOM sponge layer uses a laplacian operator with a fixed (resolution independent) coefficient, nu_top.  This creates a severe CFL condition - if we double the resolution, we must decrease dt_vis_tom by 4x.    For a 72L configuration, hypervis_subcycle_tom (THETA) or hypervis_subcycle (PREQX) must be set so that:  
    • NE30:    dt_vis_tom <= 500     
    • NE60:    dt_vis_tom  <= 125   
    • NE120:   dt_vis_tom <= 31      
    • NE256:   dt_vis_tom <= 6.9      
    • NE512:   dt_vis_tom <= 1.7
    • NE1024:  dt_vis_tom <= 0.43 
  • Using the nu_top CFL condition printed by the code, experiments suggest the operator is stable up to S=2.89, but unstable at S=3.38.  
  • For the theta model we recommend (dependent on future testing) to take nu_top=2.5e5 up to NE30. For resolutions > NE30, choose nu_top at the CFL limit using S=1.9
  • NOTE regarding RRM grids:  The HV coefficient is strongly resolution dependent, decreasing like dx^3 or dx^3.2.  For RRM grids we thus use a tensor hyperviscosity formulation that contains this scaling and applies it locally as the grid resolution changes.  Tensor hyperviscosity is enabled by setting hypervis_scaling>0  (typically 3 or 3.2).   For the tensor hyperviscosity, the coefficients (nu, nu_div, nu_q, nu_p, nu_s) are now specified in 1/s  units that are indepdent of resolution.  For hypervis_scaling=3.2, we currently recommend setting all coefficients to 8e-8.   Currently E3SM only uses tensor hyperviscosity for RRM grids, but we should consider using it for all grids.  


Recommended timestep tuning for new grids

...

With RRM grids, the timesteps will be controlled by the highest resolution region.  So with an RRM grid with refinement down to NE120, the timesteps should be close to what we run on a uniform cubed-sphere NE120 grid.   The timesteps may need to be slightly smaller because of the deformed elements in the transition region.   With a hiqh quality RRM mesh ( Max Dinv-based element distortion metric <= 4, see Generate the Grid Mesh (Exodus) File for a new Regionally-Refined Grid) we can usually run with the expected dt_dyn and dt_tracer values, and only the viscosity coefficients need to be slightly reduced.  




Recommended settings (THETA)   

...

ResolutionTimestepsNamelist settingsNotesTested?
1 degree (NE30)

dtime=1800  (ATM_NCPL=72)

dt_remap=600
dt_tracer=1800
dt_dyn=dt_vis=dt_vis_q=300
dt_vis_tom=300



nu=1e15
nu_top = 2.5e5    

se_tstep=300





With dt_remap=1800, we see occasional (every 2-3 years) dp3d limiter activation, meaning that the model levels are approaching zero.  This appears to be due to strong divergence above tropical cyclones created by one of the parameterizations.

HS+topo(72L):  H and NH 

F-EAMv1-AQP1:  H and NH 

FAV1C-L:  H and NH


1/4 degree (NE120)

dtime=900  (ATM_NCPL=144)

dt_remap=150
dt_tracer=450..650
dt_dyn=dt_vis=dt_vis_q=75
dt_vis_tom=75

nu=1.5e13
nu_top=1e5

se_tstep=75



CFL estimates suggest:

dt_vis_tom*nu_top <= 31*2.5e5

nu_top=2.5e5 would need
hypervis_subcycle_tom=3


HS+topo(72L):  H and NH (with dt_remap=75 and theta limiter to handle unphysical boundary layer)

F-EAMv1-AQP1:  H  and NH, both 72 and 128 levels (1+ years)

FC5AV1C-H01B:  NH 72L runs several years

12km

(NE256)

dtime=480?
ATM_NCPL=180

dt_remap=80
dt_tracer=240..360
dt_dyn=dt_vis=dt_vis_q=40
dt_vis_tom = 40

nu=1.6e12
nu_top=4e4
se_tstep=40





nu_tom=4e4 is running at the code's estimate of the CFL limit with S=1.9

dt=dt_dyn/ dt_remap/ dt_tracer / dt_phys

F-EAMv1-AQP1:
H-128L running with default timesteps. (1 month)  

NH-128L:

  • tstep_type=9:  Runs with dt=37.5/75/300/600.  crashes with dt=40/40/240/480 after 6d, dp3d<0 around layer 30.   
  • with tstep_type=10: dt=40/40/240/480  ran for 1 month.  


6km (NE512)dt_remap=40
dt_tracer=120..180
dt_dyn=dt_vis=dt_vis_q=20
dt_vis_tom=20

nu=2e11
nu_top=2e4
se_tstep=20



CFL estimates suggest:

dt_vis_tom*nu_top <= 1.7*2.5e5,

nu_top=2.5e5 needs hypervis_subcycle_tom=13 

F-EAMv1-AQP1:

H-128L:  dt=20/40/120/240 good.  

NH-128L: 

  • tstep_type=9:  dt=20/40/120/240: crash bad EOS 2.4days.  dt=18.75/37.5/150/300, ran 20days.
  • tstep_type=10: dt=20/40/120/240 crash 3.1days.  bad dp3d layer 75
3km (NE1024)

dt_remap=20
dt_tracer=60..90
dt_dyn=dt_vis=dt_vis_q=10
dt_vis_tom=10

nu=2.5e10
nu_top=1e4
se_tstep=10


CFL estimates suggest:

dt_vis_tom*nu_top <= 0.43*2.5e5

with nu_top=2.5e5, 
hypervis_subcycle_tom=24



...