As part of the efforts in the CMDV project, interfaces to integrate the MOAB unstructured mesh library with the TempestRemap remapping tool have been undertaken. Detailed information on the algorithmic and implementation aspects of this effort have been written in a manuscript submitted to Geoscientific Model Development [1]. This work has led to the development of a new offline remapping tool called mbtempest, which exposes the functionality to compute the supermesh or intersection mesh between two unstructured source and target component grids, in addition to using this supermesh for computing the remapping weights to project solutions between the grids. This functionality is part of the critical worflow with E3SM, where the generated remapping weights in the offline step are consumed by MCT at runtime to seamlessly transfer solution data between components (atm↔ocn, atm↔lnd, etc).
...
| Code Block |
|---|
mpiexec -n 4 tools/mbconvert -g -o "PARALLEL=WRITE_PART" -O "PARALLEL=BCAST_DELETE" -O "PARTITION=TRIVIAL" -O "PARALLEL_RESOLVE_SHARED_ENTS" outputCSMesh.exo outputCSMesh.h5m |
...
Partitioning meshes for parallel runs
MOAB supports generating partitions for the mesh
...
using either Zoltan or Metis. The mbpart tool can be utilized to generate the PARALLEL_PARTITION tag in MOAB to enable parallel runs and computation of intersection/remapping weights in considerably reduced times.
To partition meshes on say 1024 processes, the following arguments can be used in the mbpart tool.
| Code Block |
|---|
METIS (RCB) : tools/mbpart 1024 -m ML_RB -i 1.005 outputCSMeshinput_mesh.h5m outputCSMeshoutput_mesh_p1024_mRB.h5m METIS (KWay) : tools/mbpart 1024 -m ML_KWAY outputCSMeshinput_mesh.h5m outputCSMeshoutput_mesh_p1024_mKWAY.h5m Zoltan (RCB) : tools/mbpart 1024 -z RCB -i 1.005002 outputCSMeshinput_mesh.h5m outputCSMeshoutput_mesh_p1024_zRCB.h5m Zoltan (PHG) : tools/mbpart 1024 -z PHG outputCSMeshinput_mesh.h5m outputCSMeshoutput_mesh_p1024_zPHG.h5m |
Now the mesh files outputCSMesh_p1024_*.h5m file can be used in parallel as an input for either a source or target grid. Similar procedure can also be performed on other exodus meshes to make them parallel aware in order to be used with mbtempest.
Partitioning meshes with the "inferred" strategy for better performance
A more recent update to the mbpart tool is to use the concept of inferred partitions such that the geometric locality on the source and target grids are preserved as much as possible to minimize communication at runtime during intersection mesh computation. This strategy has been shown to provide considerable speedup in the intersection mesh computation and is the preferred workflow especially when one of the grids has topological holes (OCN mesh). In order to generate the inferred partitions, we usually choose the target mesh as the primary partition and the source mesh as the secondary partition. Then, the source mesh partitions are inferred based on the target mesh partition RCB tree. The commands to generate the inferred source partitions are shown below.
| Code Block |
|---|
Inferred partitioner usage: tools/mbpart #parts -z RCB -b --scale_sphere -p 2 target_input_mesh.h5m target_output_mesh.h5m --inferred source_input_mesh.h5m
Example: tools/mbpart 1024 -z RCB -b --scale_sphere -p 2 ocean.oEC60to30v3.h5m ocean.oEC60to30v3.p1024.h5m --inferred NE1024.h5m |
The inferred source partition mesh is written out to source_input_mesh_inferred.h5m. In the example above, the NE1024.h5m mesh is partitioned using the inferred strategy and written out to NE1024_inferred.h5m.
Note that only Zoltan interface in mbpart supports this strategy, and hence becomes a required dependency for MOAB to enable better partitioning for remapping.
Generating intersection meshes
...
If steps (a)-(g) pass successfully, the MOAB libraries and tools, along with interfaces for TempestRemap will be installed in $INSTALL_PREFIX/moab directory. The offline remapping weight computation tool, mbtempest, will also be installed during this process and can then be used standalone to generate the weight files as needed.
References
1 Mahadevan, V. S., Grindeanu, I., Jacob, R., and Sarich, J.: Improving climate model coupling through a complete mesh representation: a case study with E3SM (v1) and MOAB (v5.x), Geosci. Model Dev. Discuss., https://doi.org/10.5194/gmd-2018-280, in review, 2018.