As part of the efforts in the CMDV project, interfaces to integrate the MOAB unstructured mesh library with the TempestRemap remapping tool have been undertaken. This work has led to the development of a new offline remapping tool called mbtempest, which exposes the functionality to compute the supermesh or intersection mesh between two unstructured source and target component grids, in addition to using this supermesh for computing the remapping weights to project solutions between the grids. This functionality is part of the critical worflow with E3SM, where the generated remapping weights in the offline step are consumed by MCT at runtime to seamlessly transfer solution data between components (atm↔ocn, atm↔lnd, etc).
Getting started
In order to build the MOAB-TempestRemap stack with parallel MPI launch support, we suggest the following list of commands. First define an installation prefix directory where the stack of library, includes and tools will be installed. Let us call this as the $INSTALL_PREFIX environment variable.
Dependencies
Before getting started, for your architecture of choice, whether that is your laptop or a LCF machine, create a list of following compatible environment variables that can be used to build the stack.
- MPI-enabled C, C++, and Fortran compiler wrappers that are exported in the local environment as
$CC,$CXX, and$FC. - Next, verify installations of dependent libraries such as
$HDF5_DIRand$NETCDF_DIRthat have been compiled with MPI support using the$CC, $CXX, $FCcompilers. - Get Eigen3 package from the webpage and untar to the
$INSTALL_PREFIX/eigen3directory with the following command
a. Download: wget https://bitbucket.org/eigen/eigen/get/3.3.7.tar.gz OR curl https://bitbucket.org/eigen/eigen/get/3.3.7.tar.gz -O
b. Move:mv eigen-eigen* $INSTALL_PREFIX/eigen3
Build
To get the entire stack working correctly, we need to find parallel-enabled dependency installations for HDF5 and NetCDF that are built with MPI library support for the current architecture
TempestRemap
a. Clone repository: `git clone https://github.com/ClimateGlobalChange/tempestremap.git tempestremap`
b. Create build dir:cd tempestremap && mkdir build
c. Generate configure script:autoreconf -fi
d. Go to build dir:cd build
e. Configure:../configure --prefix=$INSTALL_PREFIX/tempestremap --with-netcdf=$NETCDF_DIR --with-hdf5=$HDF5_DIR CC=$CC CXX=$CXX CXXFLAGS="-g -O2"
f. Build and install:make all && make install
At the end of this series of steps, the TempestRemap libraries and tools (GenerateCSMesh, GenerateICOMesh, GenerateOverlapMesh, GenerateOfflineMap among others) will be installed in$INSTALL_PREFIX/tempestremapdirectory.- MOAB
a. Clone repository: `git clone https://bitbucket.org/fathomteam/moab.git moab`
b. Checkout feature branch:git checkout vijaysm/tempest-master-API
c. Create build dir:cd moab && mkdir build
d. Generate configure script:autoreconf -fi
e. Go to build dir:cd build
f. Configure:../configure --prefix=$INSTALL_PREFIX/moab --with-mpi --with-tempestremap=$INSTALL_PREFIX/tempestremap --with-netcdf=$NETCDF_DIR --with-hdf5=$HDF5_DIR CC=$CC FC=$FC F77=$FC CXX=$CXX CXXFLAGS="-g -O2" --with-eigen3=$INSTALL_PREFIX/eigen3
g. Build and install:make all && make install
If steps (a)-(g) pass successfully, the MOAB libraries and tools, along with interfaces for TempestRemap will be installed in$INSTALL_PREFIX/moabdirectory. The offline remapping weight computation tool,mbtempest, will also be installed during this process and can then be used standalone to generate the weight files as needed.
Using the mbtempest tool
The mbtempest tool exposes the algorithms and interfaces to invoke TempestRemap through MOAB to generate remapping weights for combinations of discretizations and unstructured meshes defined on a source grid to a target grid. Most of the options supported by TempestRemap tools are provided in this unified interface with one key difference: the entire workflow makes use of MPI parallelism. This implies that the overlap mesh computation that would normally use the GenerateOverlapMesh tool in TempestRemap would be replaced by the MOAB's parallel implementation of an advancing front intersection computation. This intersection mesh can subsequently be written out to file as an intermediate step or used by mbtempest to generate the remapping weights needed to project a solution field from source to target component grid. Prescriptions to preserve conservation of scalar/flux data and to impose monotonicity constraints are also available as options to pass to the mbtempest tool.
[bash]> tools/mbtempest -h Usage: mbtempest --help | [options] Options: -h [--help] : Show full help text -t [--type] <int> : Type of mesh (default=CS; Choose from [CS=0, RLL=1, ICO=2, OVERLAP_FILES=3, OVERLAP_MEMORY=4, OVERLAP_MOAB=5]) -r [--res] <int> : Resolution of the mesh (default=5) -d [--dual] : Output the dual of the mesh (generally relevant only for ICO mesh) -w [--weights] : Compute and output the weights using the overlap mesh (generally relevant only for OVERLAP mesh) -c [--noconserve] : Do not apply conservation to the resultant weights (relevant only when computing weights) -v [--volumetric] : Apply a volumetric projection to compute the weights (relevant only when computing weights) -n [--monotonic] <int>: Ensure monotonicity in the weight generation -l [--load] <arg> : Input mesh filenames (a source and target mesh) -o [--order] <int>: Discretization orders for the source and target solution fields -m [--method] <arg>: Discretization method for the source and target solution fields -g [--global_id] <arg>: Tag name that contains the global DoF IDs for source and target solution fields -f [--file] <arg> : Output remapping weights filename -i [--intx] <arg> : Output TempestRemap intersection mesh filename
Generating component grids with mbtempest
The mbtempest tool can be used to generate CS (cubed-sphere), RLL (lat/lon), ICO (triangular) and polygonal (MPAS-like) meshes through appropriate invocation with the -t [--type] argument in combination with -r [--res] option. Note that these mesh generation workflows are only run in serial and a handoff to TempestRemap through its public API is done internally. Some examples for these runs are provided below.
Cubed-Sphere meshes
[bash]> tools/mbtempest -t 0 -r 120 -f outputCSMesh.exo Creating TempestRemap Mesh object ... ========================================================= ..Generating mesh with resolution [120] ..Writing mesh to file [outputCSMesh.exo] Nodes per element ..Block 1 (4 nodes): 86400 ..Mesh generator exited successfully ========================================================= [LOG] Time taken to create Tempest mesh: max = 0.0571213, avg = 0.0571213 [bash]> tools/mbsize outputCSMesh.exo File outputCSMesh.exo: type count total minimum average rms maximum std.dev. ------- ------ ------- ---------------------------------- ---------------------------------- ---------------------------------- ---------------------------------- ---------------------------------- Quad 86400 13 0.00012195 0.00014544 0.00014602 0.00017133 1.3029e-05 1D Side 345600 4.2e+03 0.0092562 0.01218 0.012217 0.01309 0.00094931 Vertex 86402
RLL meshes
[bash]> tools/mbtempest -t 1 -r 50 -f outputRLLMesh.exo Creating TempestRemap Mesh object ... longitude_edges = [0, 3.6, 7.2, 10.8, 14.4, 18, 21.6, 25.2, 28.8, 32.4, 36, 39.6, 43.2, 46.8, 50.4, 54, 57.6, 61.2, 64.8, 68.4, 72, 75.6, 79.2, 82.8, 86.4, 90, 93.6, 97.2, 100.8, 104.4, 108, 111.6, 115.2, 118.8, 122.4, 126, 129.6, 133.2, 136.8, 140.4, 144, 147.6, 151.2, 154.8, 158.4, 162, 165.6, 169.2, 172.8, 176.4, 180, 183.6, 187.2, 190.8, 194.4, 198, 201.6, 205.2, 208.8, 212.4, 216, 219.6, 223.2, 226.8, 230.4, 234, 237.6, 241.2, 244.8, 248.4, 252, 255.6, 259.2, 262.8, 266.4, 270, 273.6, 277.2, 280.8, 284.4, 288, 291.6, 295.2, 298.8, 302.4, 306, 309.6, 313.2, 316.8, 320.4, 324, 327.6, 331.2, 334.8, 338.4, 342, 345.6, 349.2, 352.8, 356.4, 360] latitude_edges = [-90, -86.4, -82.8, -79.2, -75.6, -72, -68.4, -64.8, -61.2, -57.6, -54, -50.4, -46.8, -43.2, -39.6, -36, -32.4, -28.8, -25.2, -21.6, -18, -14.4, -10.8, -7.2, -3.6, 0, 3.6, 7.2, 10.8, 14.4, 18, 21.6, 25.2, 28.8, 32.4, 36, 39.6, 43.2, 46.8, 50.4, 54, 57.6, 61.2, 64.8, 68.4, 72, 75.6, 79.2, 82.8, 86.4, 90] ..Generating mesh with resolution [100, 50] ..Longitudes in range [0, 360] ..Latitudes in range [-90, 90] ..Writing mesh to file [outputRLLMesh.exo] Nodes per element ..Block 1 (4 nodes): 5000 ..Mesh generator exited successfully ========================================================= [LOG] Time taken to create Tempest mesh: max = 0.016014, avg = 0.016014 [bash]> tools/mbsize outputRLLMesh.exo File outputRLLMesh.exo: type count total minimum average rms maximum std.dev. ------- ----- ------- ---------------------------------- ---------------------------------- ---------------------------------- ---------------------------------- ---------------------------------- Quad 5000 13 0.00012384 0.0025112 0.0027889 0.0039426 0.0012132 1D Side 20000 1e+03 0 0.051401 0.054405 0.062822 0.017829 Vertex 4902
ICO meshes
[bash]> tools/mbtempest -t 2 -r 50 -f outputICOMesh.exo
Creating TempestRemap Mesh object ...
------------------------------------------------------------
Generating Mesh.. Done
Writing Mesh to file
..Mesh size: Nodes [25002] Elements [50000]
..Nodes per element
....Block 1 (3 nodes): 50000
..Done
[LOG] Time taken to create Tempest mesh: max = 0.0272096, avg = 0.0272096
[bash]> tools/mbsize outputICOMesh.exo
File outputICOMesh.exo:
type count total minimum average rms maximum std.dev.
------- ------ ------- ---------------------------------- ---------------------------------- ---------------------------------- ---------------------------------- ----------------------------------
Tri 50000 13 0.00023314 0.0002513 0.00025155 0.00027497 1.1294e-05
1D Side 150000 3.6e+03 0.022143 0.024179 0.024216 0.027274 0.0013384
Vertex 25002
ICO-Dual (polygonal) meshes
[bash]> tools/mbtempest -t 2 -r 50 -d -f outputICODMesh.exo Creating TempestRemap Mesh object ... ------------------------------------------------------------ Generating Mesh.. Done Writing Mesh to file ..Mesh size: Nodes [50000] Elements [25002] ..Nodes per element ....Block 1 (6 nodes): 25002 ..Done [LOG] Time taken to create Tempest mesh: max = 0.0774914, avg = 0.0774914
Generating intersection meshes
Once we have a source and target grid that is to be used to compute the remapping weights, mbtempest can be used in a similar fashion to GenerateOverlapMesh/GenerateOfflineMap in TempestRemap or the ESMF_RegridWeightGen tool with ESMF to create the field projection weights. Note that unlike ESMF, there is no need to create a dual of the higher-order Spectral Element (SE) mesh using the TempestRemap workflow. Some examples with different options are provided below.
1. mpiexec -n 1 tools/mbtempest -t 5 -l outputCSMesh.exo -l outputICOMesh.exo -f moab_mbtempest_remap_csico_fvfv.h5m -i moab_intx_file2.exo -w 2. mpiexec -n 2 tools/mbtempest -t 5 -l $MOAB_SRC_DIR/MeshFiles/unittest/wholeATM_T.h5m -l $MOAB_SRC_DIR/MeshFiles/unittest/recMeshOcn.h5m -f moab_mbtempest_remap_fvfv.h5m -i moab_intx_file2.exo -w -m fv -m fv 3. mpiexec -n 4 tools/mbtempest -t 5 -l $MOAB_SRC_DIR/MeshFiles/unittest/wholeATM_T.h5m -l $MOAB_SRC_DIR/MeshFiles/unittest/recMeshOcn.h5m -f moab_mbtempest_remap_sefv.h5m -i moab_intx_file2.exo -w -m cgll -m fv -o 4 -o 1 -g GLOBAL_DOFS -g GLOBAL_ID