...
MOAB will be pre-built and installed on officially supported machines.
Building MOAB library with TPL dependencies needed for E3SM
These instructions assume you have already ported E3SM to your machine (you have it defined in config_machines.xml and can build and run cases.)
There are several option dependencies for MOAB that are required for MOAB-in-E3SM. Features and tools To build the MOAB library specifically for E3SM, the following dependencies are required and mandatory for scalable coupler workflows.MOAB has to be configured with these TPLs to work consistently with E3SM and to support both online and offline workflows. The appropriate features and tools needed within MOAB will get automatically built and enabled based on the dependencies when configuring the during MOAB libraryconfiguration.
MPI: MOAB supports both MPICH- and OpenMPI-based libraries configured externally to enable scalable mesh manipulation algorithms. Needed Required dependency for the MOAB-coupler. (Without MPI, only a serial version of MOAB will be built.) Use the same MPI library/wrappers as E3SM.
HDF5: To manage the data dependencies and to natively support parallel I/O, MOAB uses a native HDF5-based file format to represent the entire MOAB data model and this is the recommended interface and format for scalable I/O of MOAB meshes and associated data. If MPI is enabled, HDF5 should be built with parallel support. This is required because TempestRemap (below) needs it. It also HDF5 support allows you to output any E3SM state maintained by MOAB in
h5m
files which that are tremendously useful in debugging .NetCDF: MOAB library optionally depends on the NetCDF libraries (C) to compile the ExodusII reader/writer. If MPI is enabled, then the underlying HDF5 dependency for NetCDF should have parallel support i.e., serial HDF5 dependency will not pass configuration checks. If you built it for E3SM, use that oneas they can be directly visualized with VisIt using the MOAB plugin (on a sphere and with the exact mesh). If you built HDF5 for E3SM (with parallel support), use that to enable HDF5 support in MOAB configuration. For MOAB-E3SM workflows, this is an optional dependency unless you expect to develop and debug MOAB driver.Parallel-NetCDF: MOAB library optionally depends on the uses parallel-NetCDF (pnetcdf) libraries to compile the support for parallel reader and writer of NetCDF meshes library directly for reading NetCDF (v3) meshes in parallel and linear map weight files . This is a required dependency to enable parallel I/O of MPAS, SCRIP, ESMF, and Exodus meshes. Use the same one you built for E3SM.(for instance the r2o map). All other component maps are generally calculated online by the MOAB coupler. Use the same PNetCDF library as E3SM. For MOAB-E3SM workflows, this is a required dependency.
Zoltan: Support for online or offline partitioning through Zoltan (and its dependencies on Scotch, ParMetis etcoptionally configured with dependency on ParMetis/Metis) can be utilized through with the MOAB partitioner tool and at runtime (
mbpart
). The runtime partitioning can be invoked with appropriate file read options (PARTITION_METHOD=RCBZOLTAN;
). This dynamic rebalancing is important to achieve good scalability when readingnc
files . If reading parallelh5m
files, we recommend using the offline partitioning tool withmbpart
with-z RCB
to use the Recursive Bisection geometric partitioner implementation in Zoltan. This is required for E3SM’s moab couplerin parallel. For MOAB-E3SM workflows, this is a required dependency.TempestRemap: Provides support for both offline and online remapping of Climate field data on unstructured spherical meshes. RequiredThe offline remapping weights can be generated using the
mbtempest
tool. Refer to Offline remapping workflow with mbtempest for more details. For MOAB-E3SM workflows, this is a required dependency.Eigen3: A substitute for BLAS/LAPACK interfaces . Required (if not available locally). This is a required dependency for MOAB when calculating weights with TempestRemap as we store the SparseMatrix linear map internally on every task and perform SpMV using the Eigen3 data-structures. Required.interfaces. For MOAB-E3SM workflows, this is a required dependency.
Optional dependencies (not needed to run the default moab MOAB coupler)
Metis/ParMetis: MOAB can optionally use the Metis or ParMetis library for partitioning mesh files in an offline form through the MOAB’s
mbpart
tool. This may be useful to pre-partition the mesh apriori in offline remapping workflows, especially when using Metis KWAY graph partitioning algorithm (MPAS).NetCDF: MOAB library optionally depends on the NetCDF(-C) libraries to compile the ExodusII reader/writer. If MPI is enabled, then the underlying HDF5 dependency for NetCDF should have parallel support i.e., serial HDF5 dependency will not pass internal configuration checks. If you built the NetCDF library for E3SM, consistently use that installation to enable NetCDF support in MOAB.
Configuring and building MOAB from source
...
Currently, both CMake and Autotools are maintained simultaneously to support all platforms (including Windows). In this section, we will focus only on the autotools configuration system as this remains the most tested configuration approach for nowat the moment. In the future (FY24Q3/4), we will deprecate the autotools configuration system and use the CMake-based approach instead as the default. When this happens, the instructions will be updated accordingly.
...
(Below use the same compiler and mpi , MPI and I/O libraries you have set up for E3SM. If you have mutiple multiple compiler-mpi MPI options, you would will need to build MOAB for each compiler family with corresponding TPL dependencies.)
Override default compilers with environment variables or configure-time options with the compilers you use for E3SM:
CC, CXX, FC, F77
For the MPI installed for E3SM, use
--with-mpi=$MPI_DIR
For the HDF5, NetCDF, PNetCDF installed, use
--with-hdf5=$HDF5_DIR --with-netcdf=$NETCDF_DIR --with-pnetcdf=$PNETCDF_DIR
to specify these external dependencies.Zoltan must currently be installed separately. use
--with-zoltan=$ZOLTAN_DIR
For TempestRemap, if you pre-installed it, then use
--pnetcdf=$PNETCDF_DIR
To simplify the process of building MOAB TPLs that are not used directly by E3SM, we currently offer a workflow to automatically pull the TPL sources and build it on the machine. These packages will be installed in the same location as the specified MOAB prefix (
INSTALL_DIR
).
Dependency | Pre-built Configuration | Auto-download Configuration |
---|---|---|
Zoltan |
|
|
Eigen3 |
|
|
TempestRemap |
|
...
|
If you built NetCDF4 and HDF5, add
--with-hdf5=$HDF5_DIR --with-netcdf=$NETCDF_DIR
Execute the
configure
script with desired configuration options either in-source or out-of-source (recommended) build directory. A typical MOAB configuration command with all TPLs enabled for E3SM pre-built for your machine might look like the one shown below, where$PACKAGE_DIR
indicates the location of the installation prefix for the particular TPL package.Code Block $MOAB_SRC/configure CC=$E3SM_MPI_DIR/bin/mpicc \ CXX=$E3SM_MPI_DIR/bin/mpicxx \ FC=$E3SM_MPI_DIR/bin/mpif90 \ F77=$E3SM_MPI_DIR/bin/mpif77 \ --prefix=$MOAB_INSTALL_DIR \ --enable-debug --enable-optimize \ --with-mpi=$MPI$E3SM_MPI_DIR \ --with-eigen3hdf5=$EIGEN3$E3SM_INCLUDEHDF5_DIR \ --with-hdf5netcdf=$HDF5$E3SM_NETCDF_DIR \ --with-netcdfpnetcdf=$NETCDF$E3SM_PNETCDF_DIR \ --with-pnetcdfeigen3=$PNETCDF$EIGEN3_INCLUDE_DIR \ --with-tempestremap=$TEMPESTREMAP_DIR \ --with-zoltan=$ZOLTAN_DIR
However, if external libraries like Eigen3, TempestRemap, and Zoltan are to be auto-downloaded and configured dynamically, then one can use the following command.
Code Block |
---|
$MOAB_SRC/configure CC=$E3SM_MPI_DIR/bin/mpicc \
CXX=$E3SM_MPI_DIR/bin/mpicxx \
FC=$E3SM_MPI_DIR/bin/mpif90 \
F77=$E3SM_MPI_DIR/bin/mpif77 \
--prefix=$MOAB_INSTALL_DIR \
--enable-debug --enable-optimize \
--with-mpi=$E3SM_MPI_DIR \
--with-hdf5=$E3SM_HDF5_DIR \
--with-netcdf=$E3SM_NETCDF_DIR \
--with-pnetcdf=$E3SM_PNETCDF_DIR \
--download-eigen3 \
--download-tempestremap \
--download-zoltan |
Once configuration with the autotools configuration workflow is complete in the build directory, run the following commands to build the library and verify the build.
Compile MOAB and supported tools:
make -j4
Verify configuration and build setup:
make check
...
(Some checks may not pass without HDF5. Ignore those).
Note: If running on machines that require a job submission, request an interactive node before you run
make check
To install the compiled libraries, headers, and tools in the
prefix
directory, run:make install
If you experience any issues during the configuration or linking phases of MOAB, please email the mailing list: moab-dev@mcs.anl.gov , or contact Vijay M , Iulian Grindeanu or #coupler on Slack for additional help.
Once you have verified MOAB has been built correctly, you need to tell update the CIME machine configuration to specify the MOAB_ROOT
(point to the installation prefix).
Edit the entry for your machine config that it is availablein config_machines.xml
and make changes like below (example from pm-cpu configuration)
Code Block |
---|
<environment_variables compiler="intel">
<env name="MOAB_ROOT">$SHELL{if [ -z "$MOAB_ROOT" ]; then echo /global/cfs/cdirs/e3sm/software/moab/intel; else echo "$MOAB_ROOT"; fi}</env>
</environment_variables> |
In the above example, replace the MOAB_ROOT
path with your MOAB installation prefix directory. In the above case, MOAB was built for multiple compilers so “intel” is in the path and there’s another entry for “gnu” on Perlmutter.
After you’ve built MOAB and modified config_machines.xml
, you can try building and running a case.
A simple create_newcase line is:
Code Block |
---|
./create_newcase --case Wne4mbtest --res ne4pg2_oQU480 --compset WCYCL1850NS --driver moab |
You can add --driver moab
to any B or F case at this time.
For create_test, a command like the following:
Code Block |
---|
./create_test SMS_Vmoab.ne4pg2_oQU480.WCYCL1850NS |
You can add “Vmoab” to any watercycle or F case test.
Pre-built recipes (Conda)
MOAB comes pre-built as part of a conda package to use offline tools like mbtempest. DO NOT use the conda installation for of MOAB for the MOAB coupler in E3SM as often the preinstalled package will be suboptimal for the cluster or machine in use. However, such installations are useful for offline remapping (mbtempest) and computational analysis workflows (with Python interfaces) using MOAB (and pyMOABPyMOAB).
Once you have a conda environment setup, use the command below to install MOAB with TempestRemap enabled (with or without MPI).
...