Running the ACME model requires a layout for which model cores are assigned to handle which model components called the PE layout. There are currently only a few people who know how to do this and there is no documentation of the process. This is a huge bottleneck which makes running on a new machine or coming up with an efficient layout for a new compset slow. The goal of this page is to provide the info needed for anyone on the project to create their own PE layouts (or at least know when their layout is bad).
...
- Use the same decomposition for all components
- Following the guidelines above for Atmosphere load balancing, find the best decomposition based on ATM run time (ignore other components initially)
- Compare ATM run time vs. TOT run time when there are few MPI tasks (lots of work per node). ATM should be most of the cost. Lets say ATM is 90% of the cost
- As you increase the number of nodes to speed up the atmosphere, monitor the ATM/TOT cost ratio. If it starts to decrease, see if you can determine if one of the other components is starting to slow down because it is using too many cores. The TOT cost could also slow down if the cost to couple becomes large, but this cost is very hard to isolate and interpret. This step requires experience and trial and error. Some strategies to consider from Patrick Worley (Unlicensed):
- Put all of the data models on their own nodes and leave these fixed (don't scale) as you scale up ATM and CPL and LND. This has been used especially when memory is limited.
- ROF is a special case - it does not scale well, so you often don't want to scale this up as fast. If LND is cheap, you can stop this early as well. And CPL may run out of scaling at some point as well.
About Parallel I/O (PIO) Layout:
Input/output (IO) can also have a big impact on model performance. To avoid convolving I/O performance with model performance, the usual steps for optimizing PE layout are:
...