Make a new graph partition for MPAS
When you need to make a new mpas*.part.N file because you changed the number of nodes an MPAS component runs on. These instructions including adding the file to the repo for others to use.
Step-by-step guide for MPAS-Ocean
- See if "metis" or "gpmetis" is available on your platform and add it to your path.
- On Cray systems, gpmetis is available after `module load metis`.
- Do "module spider metis" to find the module to load on your system.
- Install it from conda
- Find the base mpas graph file called something like "mpas-o.graph.info.151031". These files should be in inputdata/ocn/mpas-o respectively.
- Run the metis command with the last argument being the number of nodes for the partition.
- Example: "gpmetis mpas-cice.graph.info.151010 120" will generate a *.part.120 file for mpas-cice.
- Place the new file in the correct inputdata subdirectory for the model to find it.
Please do not do this for MPAS-Seaice, as the resulting graph files are not well load balanced (many cores are wasted on equatorial regions without sea ice).
Making better load balanced graph files for MPAS-Seaice
Maybe ask for help from the MPAS team. But here are detailed instructions for those who want to give it a go:
http://mpas-dev.github.io/MPAS-Tools/stable/seaice/partition.html
MPAS-Seaice is rather poorly load balanced with standard MPAS graph partitions. This is because sea ice only exists in a very limited part of the domain. A script has been written that can generate somewhat better partitions. Each block is split into 3 non-contiguous sections: One for each polar region and a larger one for the equatorial region. For highly desired changes to this script contact Adrian Turner or Xylar Asay-Davis.
Create a conda environment with
mpas_tools
and the no-MPI version of ESMF:mamba create -y -n mpas_tools python=3.11 "mpas_tools>=0.20.0" "esmf=8.2.0=nompi*" mamba activate mpas_tools
Run the sea ice graph generation tools
cd /lcrc/group/e3sm/public_html/inputdata/ice/mpas-seaice/<mesh_name> simple_seaice_partitions -m <mesh_file> -p partitions/mpas-seaice.graph.info.<creation_date> -n <core_count> <core_count> <core_count>
If you are not on LCRC, you need to download the required input files somewhere:
https://web.lcrc.anl.gov/public/e3sm/mpas_standalonedata/mpas-seaice/partition/seaice_QU60km_polar.nc
https://web.lcrc.anl.gov/public/e3sm/mpas_standalonedata/mpas-seaice/partition/icePresent_QU60km_polar.nc
and then point to them with the-d
.
Group read/write and world read permission are added automatically, so you should not need to chmod the files.
The mesh file is a file with the mesh you are trying to partition, typically seaice.*.nc
or mpassi.*.nc
. The creation date needs to match the creation date of other sea ice partition files in the partitions
directory (or it won't be found). You can provide multiple task counts in one call to simple_seaice_partitions
, which will be much faster than calling that tool multiple times. The tool should take care of setting permissions as well.