As part of the efforts in the CMDV project, interfaces to integrate the MOAB unstructured mesh library with the TempestRemap remapping tool have been undertaken. Detailed information on the algorithmic and implementation aspects of this effort have been written in a manuscript submitted to Geoscientific Model Development [1]. This work has led to the development of a new offline remapping tool called mbtempest, which exposes the functionality to compute the supermesh or intersection mesh between two unstructured source and target component grids, in addition to using this supermesh for computing the remapping weights to project solutions between the grids. This functionality is part of the critical worflow with E3SM, where the generated remapping weights in the offline step are consumed by MCT at runtime to seamlessly transfer solution data between components (atm↔ocn, atm↔lnd, etc).
...
The meshes generated with TempestRemap are output in the Exodus format, which is not in the natively parallel (optimized), I/O format in used by MOAB. So for cases with high resolution meshes (such as NE512 or NE120 or greater i.e., NE256/NE512/NE1024) that need to be used for computation of remapping weights, it is important that a preprocessing step is performed to partition the mesh in h5m format in order to minimize the mbtempest execution time. Such a workflow can be implemented as below.
...
Convert Exodus mesh to MOAB HDF5 (h5m) format
| Code Block |
|---|
mpiexec -n 4 tools/mbconvert -g -o "PARALLEL=WRITE_PART" -O "PARALLEL=BCAST_DELETE" -O "PARTITION=TRIVIAL" -O "PARALLEL_RESOLVE_SHARED_ENTS" outputCSMesh.exo outputCSMesh.h5m |
...
MOAB supports generating partitions for the mesh using either Zoltan or Metis. The mbpart tool can be utilized to generate the PARALLEL_PARTITION tag in MOAB to enable parallel runs and computation of intersection/remapping weights in considerably reduced times.
To partition meshes on say 1024 processes, the following arguments can be used in the mbpart tool.
...
Note that for the MOAB mbtempest workflows, we prefer Zoltan as our primary partitioner since there is an ongoing NGD effort with Dr. Karen Devine (Karen Devine (Unlicensed)), who is the PI of Zoltan partitioning library. However, for simplicity, Metis partitioner would work as well even if computational scaling is slightly sub-optimal.
To partition MOAB H5M meshes for running on say 1024 processes, the following arguments can be used in the mbpart tool.
| Code Block |
|---|
METIS (RCB) : tools/mbpart 1024 -m ML_RB -i 1.005 input_mesh.h5m output_mesh_p1024_mRB.h5m
METIS (KWay) : tools/mbpart 1024 -m ML_KWAY input_mesh.h5m output_mesh_p1024_mKWAY.h5m
Zoltan (RCB) : tools/mbpart 1024 -z RCB -i 1.002 input_mesh.h5m output_mesh_p1024_zRCB.h5m
Zoltan (PHG) : tools/mbpart 1024 -z PHG input_mesh.h5m output_mesh_p1024_zPHG.h5m |
...
Partitioning meshes with the "inferred" strategy for better performance
A more recent update to the mbpart tool is to use the concept of inferred partitions such that the geometric locality on the source and target grids are preserved as much as possible to minimize communication at runtime during intersection mesh computation. This strategy has been shown to provide considerable speedup in the intersection mesh computation, and is the preferred workflow now our preferred partitioning strategy in offline workflows, especially when one of the grids has topological holes (OCN mesh). In order to generate the inferred partitions, we usually choose the target mesh as the primary partition and the source mesh as the secondary partition. Then, the source mesh partitions are "inferred" based on the target mesh partition RCB tree. The commands to generate the inferred source partitions are shown below.
| Code Block | |||||
|---|---|---|---|---|---|
| Inferred partitioner usage
| ||||
Usage: tools/mbpart #parts -z RCB -b --scale_sphere -p 2 target_input_mesh.h5m target_output_mesh.h5m --inferred source_input_mesh.h5m
Example: tools/mbpart 1024 -z RCB -b --scale_sphere -p 2 ocean.oEC60to30v3.h5m ocean.oEC60to30v3.p1024.h5m --inferred NE1024.h5m |
...
Note that only Zoltan interface in mbpart supports supports this strategy, and hence becomes a required dependency for MOAB to enable better partitioning for remapping.
...
| Code Block |
|---|
1. mpiexec -n 1 tools/mbtempest -t 5 -l outputCSMesh.exoh5m -l outputICOMesh.exoh5m -i moab_intx_cs_ico.h5m 2. mpiexec -n 16 tools/mbtempest -t 5 -l $MOAB_SRC_DIR/MeshFiles/unittest/wholeATM_Tp16.h5m -l outputICOMesh_p16.exoh5m -i moab_intx_atm_ico.h5m 3. mpiexec -n 128 tools/mbtempest -t 5 -l $MOAB_SRC_DIR/MeshFiles/unittest/wholeATM_Tp128.h5m -l $MOAB_SRC_DIR/MeshFiles/unittest/recMeshOcn_p128.h5m -i moab_intx_atm_ocn.h5m |
The first -l option specifies the source grid to load, and the second -l option specifies the target grid to load. mbtempest can load Exodus, .nc and h5m files in serial, while the parallel I/O is generally optimized only for the native HDF5 format (h5m) at the moment. So we recommend converting meshes to this format and pre-partitioning using the instructions above. The option -i specifies the output file for the intersection mesh that is computed in MOAB using the advancing front algorithm in parallel. Due to the distributed nature of the mesh in parallel runs, the h5m file cannot be directly used with TempestRemap's GenerateOfflineMap tool. However, if mbtempest is run in serial (first case above) or in parallel, the intersection mesh in Exodus mesh format can be written out using TempestRemap underneath by performing a MPI_Gather on the root process. This process does not scale due to the aggregation of the entire mesh and associated metadata for this process and we recommend using the fully parallel mbtempest workflow directly if possible.
...
Once we have a source and target grid that is to be used to compute the remapping weights, mbtempest can be used in a similar fashion to GenerateOverlapMesh/GenerateOfflineMap in TempestRemap or the ESMF_RegridWeightGen tool with ESMF in TempestRemap to create the field projection weights. Note that unlike ESMF, there is no need to create a dual of the higher-order Spectral Element (SE) mesh using the TempestRemap workflow. Some examples with different options are provided below. Note that to enable generation of weights in addition to computing the intersection mesh, the only main option to enable is -wThe remapping weights can be generated after computing the intersection mesh by specifying the -w flag.
| Code Block |
|---|
1. Serial (default: FV-FV): mpiexec -n 1 tools/mbtempest -t 5 -l outputCSMesh.exoh5m -l outputICOMesh.exoh5m -f moab_mbtempest_remap_csico_fvfv.nc -i moab_intx_file2.exo -w 2. Parallel (explicit: FV-FV): mpiexec -n 64 tools/mbtempest -t 5 -l $MOAB_SRC_DIR/MeshFiles/unittest/wholeATM_T.h5m -l $MOAB_SRC_DIR/MeshFiles/unittest/recMeshOcn.h5m -f moab_mbtempest_remap_fvfv.nc -i moab_intx_file2.exoh5m -w -m fv -m fv 3. Parallel. Parallel (SE-FV): mpiexec -n 128 tools/mbtempest -t 5 -l $MOAB_SRC_DIR/MeshFiles/unittest/wholeATM_T.h5m -l $MOAB_SRC_DIR/MeshFiles/unittest/recMeshOcn.h5m -f moab_mbtempest_remap_sefv.nc -i moab_intx_file2.exoh5m -w -m cgll -mo fv4 -og 4GLOBAL_DOFS -om 1fv -go GLOBAL_DOFS1 -g GLOBAL_ID |
These commands will generate the remapping weights by computing the intersection mesh through advancing front intersection algorithm in MOAB and then using TempestRemap to generate the weights in parallel. The computed matrix weights are then written out in parallel in the h5m format (specified through option -f). The user can also specify the field discretization type and order by using the -m and -o options. Currently, the -m option can take fv, cgll, dgll as valid options and -o is any non-zero, positive integer representing the order of the spectral or FV discretization for climate problems.
...
NOTE: In older versions of MOAB (< 5.2.1), you have to write out the map file in parallel using h5m format and then use the h5mtoscrip tool to convert to .nc file. . However, in MOAB v5.2.1 and later, we can directly write out the remap weights in the nc file format, in parallel.
Machines with mbtempest tool pre-installed
...
| Code Block |
|---|
export HDF5_USE_FILE_LOCKING=FALSE |
Anvil:
prepare Prepare environment for intel18 Intel-v18 compiler:
source /lcrc/soft/climate/moab/anvil/intel18/anvil_env.sh
MOAB_DIR=/lcrc/soft/climate/moab/anvil/intel18
TEMPESTREMAP_DIR=/lcrc/soft/climate/tempestremap/anvil/intel18
Chrysalis:
prepare
Prepare environment for intel Intel compiler:
source /lcrc/soft/climate/moab/chrysalis/intel/chrys_intel.env
MOAB_DIR=/lcrc/soft/climate/moab/chrysalis/intel
TEMPESTREMAP_DIR=/lcrc/soft/climate/tempestremap/chrysalis/intel
Note: the serial executables need to be launched with mpiexec -np 1, if on login node.
(otherwise you may get an error like :
| Code Block |
|---|
...
mbpart -h |
...
Abort(1091087) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: |
...
MPIR_Init_thread(136): |
...
MPID_Init(950).......: |
...
MPIR_pmi_init(168)...: PMI2_Job_GetId returned 14 ) |
Best A recommended best practice is to use an interactive session on a compute node, which can be requested using the following command.
srun
...
-N
...
1
...
-t
...
10
...
--pty
...
bash
Compy:
After several iterations to get the dependency stack correct with respect to compatible parallel HDF5 and NetCDF installations by sysadmins on Compy, the other required dependencies were installed successfully and verified/tested to run without issues. Now mbtempest tool is accessible from $MOAB_DIR (see below).
...
If steps (a)-(g) pass successfully, the MOAB libraries and tools, along with interfaces for TempestRemap will be installed in $INSTALL_PREFIX/moab directory. The offline remapping weight computation tool, mbtempest, will also be installed during this process and can then be used standalone to generate the weight files as needed.
References
1 Mahadevan, V. S., Grindeanu, I., Jacob, R., and Sarich, J.: Improving climate model coupling through a complete mesh representation: a case study with E3SM (v1) and MOAB (v5.x), Geosci. Model Dev. Discuss., https://doi.org/10.5194/gmd-2018-280, in review, 2018.