Breakout 1: Monday June 26th, 1-2pm, MDT

Breakout Room 2, Telluride

Model stewardship, performance and testing

...

Breakout Room 2, Telluride

Bias reduction

This breakout follows the Polar Group’s breakout Tuesday morning, where (hopefully) the Polar Group will highlight the relevant biases.

Posters

Everyone please add posters or specific topics that you would like to discuss during the breakouts

• flash talks to introduce posters in breakout sessions

  • posters can be in-person or online

  • in-person posters will have dedicated sessions for discussion

  • in-depth discussions of online poster can be arranged individually, as needed

Notes from Monday breakout

first block

Elizabeth Hunke: lot of sea ice work going on currently is in CICE consortium

• wave / sea ice interactions (maybe I got this wrong)? something about waves

• big CICE column phys merge efforts

Andrew Roberts: big sea ice biases currently …

• is this a coupling problem or are we missing important physics?

• too much NH sea ice in winter; too little sea ice in Ant. in winter and summer

• we have (arguably) one of the most advanced sea ice models out there, what are we still doing wrong?

• list of all current / ongoing / new devel work presented … Q: is anything missing from the current list?

• things we know about:

• sea ice and ocean (coupling) params. / constants are not always consistent; someone can change in one model but not the other and break coupling

• atmos. flux interation doesn’t (always? ever?) converge

• frazil scheme (in ocean?) is not mushy (but is in sea ice?)

• sophisticated radiation phys in sea ice model but coupling is very rudimentary still; could couple over 3 bands rather than 2; are currently coupling between 2 bands rather than resolving them both

• potential of very thin ice cover in v3 … how do we combat this?  

second block (performance)

sea ice perf improvements:

• easy one is reduce mesh (that is empty of ice) at low lats

• separate dynamics and column phys (EH’s understanding of this is that it results in a physics error related to ice albedo)

• EVP dynamics kernel (other groups have shown good speed up doing this)

• accelerate EVP sub cycling convergence

WWIII perf improvements

• reducing freq. of wave spectrum

• variable res

• other I might have missed

MALI perf.

• mostly stand-alone work currently; coupling w/ e3sm in early stages

• cpu good strong scaling; gpu poor strong scaling (gpus need more work, not less)

• on PM, assembly improves by ~2x with gpus vs cpus only; total solve time cpu vs cpu+gpu remains similar for now

• working now on coupled layout on PM (starting from admittedly bad initial layout on Cori)

third block (testing)

MALI

• hundreds of tests run daily / weekly

• diff architectures

• diff builds

• CDash for reporting results

• regression tests (unit tests, mnfctd solos, benchmarks, realistic stand alone configurations)

• performance tests (realistic setup and testing of perf over time; monitoring for significant wall-clock time changes over time within two stnd)  

CICE-QC testing

• determines if two diff. sims. have the same sea ice climate (not IN e3sm yet … should we / can we get it in for all to use?)

Andrew Roberts: polar developer tool

• bash script to efficiently run many e3sm sims for testing

• fulfills several recommendations of code review deep dive (e.g., testing of energy / mass conservation)

• currently set up for Chrysalis and Anvil

• command line; similar to current “run script” but with more options (?)

• PR in; latest version in ‘simulation script’ area of repo

• can generate namelist combinations and compare them (example, compare using high-frequency coupling vs. not)

• build and queue all combos at once

• can also use it to run preset test suites (e.g, e3sm developers test suite)

• “-e” option checks for energy and mass conservation on the fly