As part of the efforts in the CMDV project, interfaces to integrate the MOAB unstructured mesh library with the TempestRemap remapping tool have been undertaken. Detailed information on the algorithmic and implementation aspects of this effort have been written in a manuscript submitted to Geoscientific Model Development [1]. This work has led to the development of a new offline remapping tool called mbtempest, which exposes the functionality to compute the supermesh or intersection mesh between two unstructured source and target component grids, in addition to using this supermesh for computing the remapping weights to project solutions between the grids. This functionality is part of the critical worflow with E3SM, where the generated remapping weights in the offline step are consumed by MCT at runtime to seamlessly transfer solution data between components (atm↔ocn, atm↔lnd, etc).
...
Partitioning meshes with the "inferred" strategy for better performance
A more recent update /wiki/spaces/ED/pages/2208170181 to the mbpart tool is to use the concept of inferred partitions such that the geometric locality on the source and target grids are preserved as much as possible to minimize communication at runtime during intersection mesh computation. This strategy has been shown to provide considerable speedup in the intersection mesh computation, and is now our preferred partitioning strategy in offline workflows, especially when one of the grids has topological holes (OCN mesh). In order to generate the inferred partitions, we usually choose the target mesh as the primary partition and the source mesh as the secondary partition. Then, the source mesh partitions are "inferred" based on the target mesh partition RCB tree. The commands to generate the inferred source partitions are shown below.
...
The environment settings for running mbtempest on comps are listed below, and stored in the file: /compyfs/software/mbtempest.envs.sh for reference.
This is version 5e41106dc9a2 from MOAB (>5.3.0) and version 72df14282a2e9 from tempestremap (>2.1.0)
| Code Block |
|---|
module load cmake/3.11.4 intel/19.0.3 mvapich2/2.3.1 pnetcdf/1.9.0 mkl/2019u3 metis/5.1.0 export MPI_DIR=/share/apps/mvapich2/2.3.1/intel/19.0.3 export METIS_DIR=/share/apps/metis/5.1.0 export EIGEN3_DIR=/share/apps/eigen3/3.3.7/include/eigen3 export HDF5_DIR=/share/apps/netcdf-MPI/intel/19.0.5/mvapich2/2.3.2 export NETCDF_DIR=/share/apps/netcdf-MPI/intel/19.0.5/mvapich2/2.3.2 export PNETCDF_DIR=/share/apps/pnetcdf/1.9.0/intel/19.0.3/mvapich2/2.3.1 export ZOLTAN_DIR=/compyfs/software/zoltan/3.83/intel/19.0.3 export TEMPESTREMAP_DIR=/compyfs/software/tempestremap/intel/19.0.3 export MOAB_DIR=/compyfs/software/moab/intel/19.0.3 |
...
If steps (a)-(g) pass successfully, the MOAB libraries and tools, along with interfaces for TempestRemap will be installed in $INSTALL_PREFIX/moab directory. The offline remapping weight computation tool, mbtempest, will also be installed during this process and can then be used standalone to generate the weight files as needed.
References
1 Mahadevan, V. S., Grindeanu, I., Jacob, R., and Sarich, J.: Improving climate model coupling through a complete mesh representation: a case study with E3SM (v1) and MOAB (v5.x), Geosci. Model Dev. Discuss., https://doi.org/10.5194/gmd-2018-280, in review, 2018.