As part of the efforts in the CMDV project, interfaces to integrate the MOAB unstructured mesh library with the TempestRemap remapping tool have been undertaken. Detailed information on the algorithmic and implementation aspects of this effort have been written in a manuscript submitted to Geoscientific Model Development [1]. This work has led to the development of a new offline remapping tool called mbtempest, which exposes the functionality to compute the supermesh or intersection mesh between two unstructured source and target component grids, in addition to using this supermesh for computing the remapping weights to project solutions between the grids. This functionality is part of the critical worflow with E3SM, where the generated remapping weights in the offline step are consumed by MCT at runtime to seamlessly transfer solution data between components (atm↔ocn, atm↔lnd, etc).
Table of Contents
Getting started
In order to build the MOAB-TempestRemap stack with parallel MPI launch support, we suggest the following list of commands. First define an installation prefix directory where the stack of library, includes and tools will be installed. Let us call this as the $INSTALL_PREFIX environment variable.
Dependencies and pre-requisites
Before getting started, for your architecture of choice, whether that is your laptop or a LCF machine, create a list of following compatible environment variables that can be used to build the stack.
- MPI-enabled C, C++, and Fortran compiler wrappers that are exported in the local environment as
$CC,$CXX, and$FC. - Next, verify installations of dependent libraries such as
$HDF5_DIRand$NETCDF_DIRthat have been compiled with MPI support using the$CC, $CXX, $FCcompilers. - Get Eigen3 package from the webpage and untar to the
$INSTALL_PREFIX/eigen3directory with the following command- Download: wget https://bitbucket.org/eigen/eigen/get/3.3.7.tar.gz OR curl https://bitbucket.org/eigen/eigen/get/3.3.7.tar.gz -O
- Move:
mv eigen-eigen* $INSTALL_PREFIX/eigen3 - export EIGEN3_DIR=
$INSTALL_PREFIX/eigen3
Build
To get the entire (MOAB-TempestRemap) stack working correctly, we need to find parallel-enabled dependency installations for HDF5 and NetCDF that are built with MPI library support for the current architecture
TempestRemap
- Clone repository: `git clone https://github.com/ClimateGlobalChange/tempestremap.git tempestremap`
- Create build dir:
cd tempestremap && mkdir build - Generate configure script:
autoreconf -fi - Go to build dir:
cd build - Configure:
../configure --prefix=$INSTALL_PREFIX/tempestremap --with-netcdf=$NETCDF_DIR --with-hdf5=$HDF5_DIR CC=$CC CXX=$CXX CXXFLAGS="-g -O2" - Build and install:
make all && make install
At the end of this series of steps, the TempestRemap libraries and tools (GenerateCSMesh, GenerateICOMesh, GenerateOverlapMesh, GenerateOfflineMap among others) will be installed in$INSTALL_PREFIX/tempestremapdirectory.
MOAB
- Clone repository: `git clone https://bitbucket.org/fathomteam/moab.git moab`
- Checkout feature branch:
git checkout vijaysm/tempest-master-API - Create build dir:
cd moab && mkdir build - Generate configure script:
autoreconf -fi - Go to build dir:
cd build - Configure:
../configure --prefix=$INSTALL_PREFIX/moab --with-mpi --with-tempestremap=$INSTALL_PREFIX/tempestremap --with-netcdf=$NETCDF_DIR --with-hdf5=$HDF5_DIR CC=$CC FC=$FC F77=$FC CXX=$CXX CXXFLAGS="-g -O2" --with-eigen3=$EIGEN3_DIR - Build and install:
make all && make install
If steps (a)-(g) pass successfully, the MOAB libraries and tools, along with interfaces for TempestRemap will be installed in$INSTALL_PREFIX/moabdirectory. The offline remapping weight computation tool,mbtempest, will also be installed during this process and can then be used standalone to generate the weight files as needed.
A simpler, consolidated build process
Or to combine both the builds together, we recommend users to go ahead with a consolidated configuration process for MOAB, which combines the configuration for TempestRemap as part of the MOAB configuration. Notice the --download-tempestremap=master option below in the configure line that instructs MOAB to clone the master branch of TempestRemap and build the dependency with $HDF5_DIR and $NETCDF_DIR specified by the user along with consistent compiler options.
MOAB and TempestRemap
a. Clone repository: `git clone https://bitbucket.org/fathomteam/moab.git moab`
b. Checkout feature branch: git checkout vijaysm/tempest-master-API
c. Create build dir: cd moab && mkdir build
d. Generate configure script: autoreconf -fi
e. Go to build dir: cd build
f. Configure: ../configure --prefix=$INSTALL_PREFIX/moab --with-mpi --download-tempestremap=master --with-netcdf=$NETCDF_DIR --with-hdf5=$HDF5_DIR CC=$CC FC=$FC F77=$FC CXX=$CXX CXXFLAGS="-g -O2" --with-eigen3=$EIGEN3_DIR
g. Build and install: make all && make install
If steps (a)-(g) pass successfully, the MOAB libraries and tools, along with interfaces for TempestRemap will be installed in $INSTALL_PREFIX/moab directory. The offline remapping weight computation tool, mbtempest, will also be installed during this process and can then be used standalone to generate the weight files as needed.
Using the mbtempest tool
The mbtempest tool exposes the algorithms and interfaces to invoke TempestRemap through MOAB to generate remapping weights for combinations of discretizations and unstructured meshes defined on a source grid to a target grid. Most of the options supported by TempestRemap tools are provided in this unified interface with one key difference: the entire workflow makes use of MPI parallelism. This implies that the overlap mesh computation that would normally use the GenerateOverlapMesh tool in TempestRemap would be replaced by the MOAB's parallel implementation of an advancing front intersection computation. This intersection mesh can subsequently be written out to file as an intermediate step or used by mbtempest to generate the remapping weights needed to project a solution field from source to target component grid. Prescriptions to preserve conservation of scalar/flux data and to impose monotonicity constraints are also available as options to pass to the mbtempest tool.
[bash]> tools/mbtempest -h Usage: mbtempest --help | [options] Options: -h [--help] : Show full help text -t [--type] <int> : Type of mesh (default=CS; Choose from [CS=0, RLL=1, ICO=2, OVERLAP_FILES=3, OVERLAP_MEMORY=4, OVERLAP_MOAB=5]) -r [--res] <int> : Resolution of the mesh (default=5) -d [--dual] : Output the dual of the mesh (generally relevant only for ICO mesh) -w [--weights] : Compute and output the weights using the overlap mesh (generally relevant only for OVERLAP mesh) -c [--noconserve] : Do not apply conservation to the resultant weights (relevant only when computing weights) -v [--volumetric] : Apply a volumetric projection to compute the weights (relevant only when computing weights) -n [--monotonic] <int>: Ensure monotonicity in the weight generation -l [--load] <arg> : Input mesh filenames (a source and target mesh) -o [--order] <int>: Discretization orders for the source and target solution fields -m [--method] <arg>: Discretization method for the source and target solution fields -g [--global_id] <arg>: Tag name that contains the global DoF IDs for source and target solution fields -f [--file] <arg> : Output remapping weights filename -i [--intx] <arg> : Output TempestRemap intersection mesh filename
Generating component grids with mbtempest
The mbtempest tool can be used to generate CS (cubed-sphere), RLL (lat/lon), ICO (triangular) and polygonal (MPAS-like) meshes through appropriate invocation with the -t [--type] argument in combination with -r [--res] option. Note that these mesh generation workflows are only run in serial and a handoff to TempestRemap through its public API is done internally. Some examples for these runs are provided below.
Cubed-Sphere meshes
[bash]> tools/mbtempest -t 0 -r 120 -f outputCSMesh.exo Creating TempestRemap Mesh object ... ========================================================= ..Generating mesh with resolution [120] ..Writing mesh to file [outputCSMesh.exo] Nodes per element ..Block 1 (4 nodes): 86400 ..Mesh generator exited successfully ========================================================= [LOG] Time taken to create Tempest mesh: max = 0.0571213, avg = 0.0571213 [bash]> tools/mbsize outputCSMesh.exo File outputCSMesh.exo: type count total minimum average rms maximum std.dev. ------- ------ ------- ---------------------------------- ---------------------------------- ---------------------------------- ---------------------------------- ---------------------------------- Quad 86400 13 0.00012195 0.00014544 0.00014602 0.00017133 1.3029e-05 1D Side 345600 4.2e+03 0.0092562 0.01218 0.012217 0.01309 0.00094931 Vertex 86402
RLL meshes
[bash]> tools/mbtempest -t 1 -r 50 -f outputRLLMesh.exo Creating TempestRemap Mesh object ... longitude_edges = [0, 3.6, 7.2, 10.8, 14.4, 18, 21.6, 25.2, 28.8, 32.4, 36, 39.6, 43.2, 46.8, 50.4, 54, 57.6, 61.2, 64.8, 68.4, 72, 75.6, 79.2, 82.8, 86.4, 90, 93.6, 97.2, 100.8, 104.4, 108, 111.6, 115.2, 118.8, 122.4, 126, 129.6, 133.2, 136.8, 140.4, 144, 147.6, 151.2, 154.8, 158.4, 162, 165.6, 169.2, 172.8, 176.4, 180, 183.6, 187.2, 190.8, 194.4, 198, 201.6, 205.2, 208.8, 212.4, 216, 219.6, 223.2, 226.8, 230.4, 234, 237.6, 241.2, 244.8, 248.4, 252, 255.6, 259.2, 262.8, 266.4, 270, 273.6, 277.2, 280.8, 284.4, 288, 291.6, 295.2, 298.8, 302.4, 306, 309.6, 313.2, 316.8, 320.4, 324, 327.6, 331.2, 334.8, 338.4, 342, 345.6, 349.2, 352.8, 356.4, 360] latitude_edges = [-90, -86.4, -82.8, -79.2, -75.6, -72, -68.4, -64.8, -61.2, -57.6, -54, -50.4, -46.8, -43.2, -39.6, -36, -32.4, -28.8, -25.2, -21.6, -18, -14.4, -10.8, -7.2, -3.6, 0, 3.6, 7.2, 10.8, 14.4, 18, 21.6, 25.2, 28.8, 32.4, 36, 39.6, 43.2, 46.8, 50.4, 54, 57.6, 61.2, 64.8, 68.4, 72, 75.6, 79.2, 82.8, 86.4, 90] ..Generating mesh with resolution [100, 50] ..Longitudes in range [0, 360] ..Latitudes in range [-90, 90] ..Writing mesh to file [outputRLLMesh.exo] Nodes per element ..Block 1 (4 nodes): 5000 ..Mesh generator exited successfully ========================================================= [LOG] Time taken to create Tempest mesh: max = 0.016014, avg = 0.016014 [bash]> tools/mbsize outputRLLMesh.exo File outputRLLMesh.exo: type count total minimum average rms maximum std.dev. ------- ----- ------- ---------------------------------- ---------------------------------- ---------------------------------- ---------------------------------- ---------------------------------- Quad 5000 13 0.00012384 0.0025112 0.0027889 0.0039426 0.0012132 1D Side 20000 1e+03 0 0.051401 0.054405 0.062822 0.017829 Vertex 4902
ICO meshes
[bash]> tools/mbtempest -t 2 -r 50 -f outputICOMesh.exo
Creating TempestRemap Mesh object ...
------------------------------------------------------------
Generating Mesh.. Done
Writing Mesh to file
..Mesh size: Nodes [25002] Elements [50000]
..Nodes per element
....Block 1 (3 nodes): 50000
..Done
[LOG] Time taken to create Tempest mesh: max = 0.0272096, avg = 0.0272096
[bash]> tools/mbsize outputICOMesh.exo
File outputICOMesh.exo:
type count total minimum average rms maximum std.dev.
------- ------ ------- ---------------------------------- ---------------------------------- ---------------------------------- ---------------------------------- ----------------------------------
Tri 50000 13 0.00023314 0.0002513 0.00025155 0.00027497 1.1294e-05
1D Side 150000 3.6e+03 0.022143 0.024179 0.024216 0.027274 0.0013384
Vertex 25002
ICO-Dual (polygonal) meshes
[bash]> tools/mbtempest -t 2 -r 50 -d -f outputICODMesh.exo Creating TempestRemap Mesh object ... ------------------------------------------------------------ Generating Mesh.. Done Writing Mesh to file ..Mesh size: Nodes [50000] Elements [25002] ..Nodes per element ....Block 1 (6 nodes): 25002 ..Done [LOG] Time taken to create Tempest mesh: max = 0.0774914, avg = 0.0774914
Note on converting Exodus meshes to parallel h5m format
The meshes generated with TempestRemap are output in the Exodus format, which is not the natively parallel (optimized) I/O format in MOAB. So for cases with high resolution meshes (such as NE512 or NE1024) that need to be used for computation of remapping weights, it is important that a preprocessing step is performed to partition the mesh in h5m format in order to minimize the mbtempest execution. Such a workflow can be implemented as below.
First convert the mesh from Exodus to HDF5 (h5m)
mpiexec -n 4 tools/mbconvert -g -o "PARALLEL=WRITE_PART" -O "PARALLEL=BCAST_DELETE" -O "PARTITION=TRIVIAL" -O "PARALLEL_RESOLVE_SHARED_ENTS" outputCSMesh.exo outputCSMesh.h5m
Next partition the mesh with Zoltan or Metis
METIS (RCB) : tools/mbpart 1024 -m ML_RB -i 1.005 outputCSMesh.h5m outputCSMesh_p1024_mRB.h5m METIS (KWay) : tools/mbpart 1024 -m ML_KWAY outputCSMesh.h5m outputCSMesh_p1024_mKWAY.h5m Zoltan (RCB) : tools/mbpart 1024 -z RCB -i 1.005 outputCSMesh.h5m outputCSMesh_p1024_zRCB.h5m Zoltan (PHG) : tools/mbpart 1024 -z PHG outputCSMesh.h5m outputCSMesh_p1024_zPHG.h5m
Now the mesh files outputCSMesh_p1024_*.h5m file can be used in parallel as an input for either a source or target grid. Similar procedure can also be performed on other exodus meshes to make them parallel aware in order to be used with mbtempest.
Generating intersection meshes
Once we have a source and target grid that is to be used to compute the remapping weights, mbtempest can be used in a similar fashion to GenerateOverlapMesh/GenerateOfflineMap in TempestRemap or the ESMF_RegridWeightGen tool with ESMF to create the field projection weights. Note that unlike ESMF, there is no need to create a dual of the higher-order Spectral Element (SE) mesh using the TempestRemap workflow. Some examples with different options are provided below.
1. mpiexec -n 1 tools/mbtempest -t 5 -l outputCSMesh.exo -l outputICOMesh.exo -i moab_intx_cs_ico.exo 2. mpiexec -n 16 tools/mbtempest -t 5 -l $MOAB_SRC_DIR/MeshFiles/unittest/wholeATM_T.h5m -l outputICOMesh.exo -i moab_intx_atm_ico.h5m 3. mpiexec -n 128 tools/mbtempest -t 5 -l $MOAB_SRC_DIR/MeshFiles/unittest/wholeATM_T.h5m -l $MOAB_SRC_DIR/MeshFiles/unittest/recMeshOcn.h5m -i moab_intx_atm_ocn.h5m
The first -l option specifies the source grid to load, and the second -l option specifies the target grid to load. mbtempest can load Exodus, .nc and h5m files in serial, while the parallel I/O is generally optimized only for the native HDF5 format (h5m). So we recommend converting meshes to this format and pre-partitioning using the instructions above. The option -i specifies the output file for the intersection mesh that is computed in MOAB using the advancing front algorithm in parallel. Due to the distributed nature of the mesh in parallel runs, the h5m file cannot be directly used with TempestRemap's GenerateOfflineMap tool. However, if mbtempest is run in serial (first case above) or in parallel, the intersection mesh in Exodus mesh format can be written out using TempestRemap underneath by performing a MPI_Gather on the root process. This process does not scale due to the aggregation of the entire mesh and associated metadata for this process and we recommend using the fully parallel mbtempest workflow directly if possible.
Generating remapping weights in parallel
Once we have a source and target grid that is to be used to compute the remapping weights, mbtempest can be used in a similar fashion to GenerateOverlapMesh/GenerateOfflineMap in TempestRemap or the ESMF_RegridWeightGen tool with ESMF to create the field projection weights. Note that unlike ESMF, there is no need to create a dual of the higher-order Spectral Element (SE) mesh using the TempestRemap workflow. Some examples with different options are provided below. Note that to enable generation of weights in addition to computing the intersection mesh, the only main option to enable is -w.
1. mpiexec -n 1 tools/mbtempest -t 5 -l outputCSMesh.exo -l outputICOMesh.exo -f moab_mbtempest_remap_csico_fvfv.h5m -i moab_intx_file2.exo -w 2. mpiexec -n 64 tools/mbtempest -t 5 -l $MOAB_SRC_DIR/MeshFiles/unittest/wholeATM_T.h5m -l $MOAB_SRC_DIR/MeshFiles/unittest/recMeshOcn.h5m -f moab_mbtempest_remap_fvfv.h5m -i moab_intx_file2.exo -w -m fv -m fv 3. mpiexec -n 128 tools/mbtempest -t 5 -l $MOAB_SRC_DIR/MeshFiles/unittest/wholeATM_T.h5m -l $MOAB_SRC_DIR/MeshFiles/unittest/recMeshOcn.h5m -f moab_mbtempest_remap_sefv.h5m -i moab_intx_file2.exo -w -m cgll -m fv -o 4 -o 1 -g GLOBAL_DOFS -g GLOBAL_ID
These commands will generate the remapping weights by computing the intersection mesh through advancing front intersection algorithm in MOAB and then using TempestRemap to generate the weights in parallel. The computed matrix weights are then written out in parallel in the h5m format (specified through option -f). The user can also specify the field discretization type and order by using the -m and -o options. Currently, the -m option can take fv, cgll, dgll as valid options and -o is any non-zero, positive integer representing the order of the spectral or FV discretization for climate problems.
Additionally, the GLOBAL_ID tag is used as an identifier to uniquely represent degrees-of-freedom (DoF) for element-based discretizations such as FV or vertex-based lower order discretizations such as FD. These options can be specified with -g options for mbtempest. However, complex discretizations should provide the unique DoF numbering in special element-based MOAB tags. In the above example (3), the GLOBAL_DOFS tag represents the SE DoF ids for the source mesh and the GLOBAL_ID tag represents that DoF ids for the target mesh, since we are computing remapping weights from SE to FV.
Note on DoF IDs for SE meshes
Currently, there is no consistent way to generate the global DoF ids for a spectral mesh in parallel, through MOAB. While the actual ID assignments for DoFs are easy in serial, the assignment in parallel could be dependent on the partitions and may not even match the numbering in HOMME. Hence, the generated mapping weights may need a different permutation to be applied before passing it back to HOMME within E3SM. So, the recommended workflow for such a case is to run HOMME to get the resolution of the grid required to generate the mapping file, and then store the DoF IDs as a GLOBAL_DOFS tag in MOAB, before being written to disk. This eliminates the need to get a consistent match when generating the weights in the MOAB mbtempest offline workflow.
Converting parallel h5m file to SCRIP file for E3SM
We are currently working on a Python script to convert the MOAB h5m file to a SCRIP file to be consumed in E3SM directly. As a future task, we will also be investing effort into writing a SCRIP file in parallel through a custom I/O interface using PNetCDF.
References
1 Mahadevan, V. S., Grindeanu, I., Jacob, R., and Sarich, J.: Improving climate model coupling through a complete mesh representation: a case study with E3SM (v1) and MOAB (v5.x), Geosci. Model Dev. Discuss., https://doi.org/10.5194/gmd-2018-280, in review, 2018.