Breakout 1: Monday June 26th, 1-2pm, MDT

Breakout Room 2, Telluride

Model stewardship, performance and testing

Breakout 2: Tuesday June 27th, 1:15pm-2:15pm, MDT

Breakout Room 2, Telluride

Bias reduction

This breakout follows the Polar Group’s breakout Tuesday morning, where (hopefully) the Polar Group will highlight the relevant biases.

Posters

Everyone please add posters or specific topics that you would like to discuss during the breakouts

• flash talks to introduce posters in breakout sessions

  • posters can be in-person or online

  • in-person posters will have dedicated sessions for discussion

  • in-depth discussions of online poster can be arranged individually, as needed

Notes from Monday breakout

sea ice model stewardship

Elizabeth Hunke: lot of sea ice work going on currently is focused on merging Icepack from the CICE consortium

• Icepack enables wave / sea ice interactions in E3SM, by providing a floe size distribution

Andrew Roberts: E3SM simulations show big sea ice biases

• is this a coupling problem or are we missing important physics?

• too much NH sea ice in winter; too little sea ice in Ant. in winter and summer

• we have (arguably) one of the most advanced sea ice models out there, what are we still doing wrong?

• list of potential physics culprits … is anything missing from the current list?

• things we know about:

  • sea ice and ocean (coupling) params. / constants are not always consistent; someone can change in one model but not the other and break coupling

  • atmos. boundary layer flux calculation doesn’t converge with increased iterations

  • frazil scheme in ocean is not mushy – the ice model takes up seawater along with newly frozen ice

  • sophisticated radiation phys in sea ice model but the coupling is still very rudimentary; could couple over 3 bands rather than 2; are currently coupling between 2 bands rather than resolving them both

  • expecting to have very thin ice cover in v3 … how do we combat this?  

performance

sea ice perf improvements:

• one is reduce mesh at lower latitudes, where ice is unlikely to appear. This is mostly done, needs a mesh to be created

• separate dynamics and column physics for coupling, so that atmosphere can run concurrently with sea ice dynamics (EH’s understanding of this is that it results in a physics error related to ice albedo being inconsistent with the final sea ice thickness distribution)

• refactor EVP dynamics into a kernel that can run on a single CPU (other groups have shown 10x speedup doing this) or GPU

• accelerate EVP subcycling convergence

WWIII perf improvements

• reducing number of frequencies used for the wave spectrum does not significantly affect accuracy

• variable resolution mesh

• port wave action source terms to GPU

MALI perf.

• mostly stand-alone work currently; coupling w/ e3sm is in early stages

• cpu good strong scaling; gpu poor strong scaling (gpus need more work, not less)

• on perlmutter, assembly improves by ~2x with gpus vs cpus only; total solve time cpu vs cpu+gpu remains similar for now

• working now on coupled layout on perlmutter (starting from admittedly bad initial layout on Cori)

testing

MALI

• hundreds of tests run daily / weekly

• diff architectures

• diff builds

• CDash for reporting results

• regression tests (unit tests, mnfctd solos, benchmarks, realistic stand alone configurations)

• performance tests (realistic setup and testing of perf over time; monitoring for significant wall-clock time changes over time within two stnd)  

CICE-QC testing

• determines if two different sims. have the same sea ice climate

• discussion PR, ready to merge to E3SM

Andrew Roberts: polar developer tool

• bash script to efficiently run many E3SM sims for testing

• fulfills several recommendations of code review deep dive (e.g., testing of energy / mass conservation)

• currently set up for Chrysalis and Anvil

• command line; similar to current “run script” but with more flexibility

• PR in; latest version in ‘simulation script’ area of repo

• can generate namelist combinations and compare them (example, compare using high-frequency coupling vs. not)

• builds all combos at once and then queues them all

• can also use it to run preset test suites (e.g, E3SM developers test suite)

• “-e” option checks for energy and mass conservation on the fly