Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 11 Next »

To use the MOAB-based coupler (E3SM/driver-moab), you will need to install MOAB and some of its dependencies on your machine to get started.

MOAB will be pre-built and installed on officially supported machines.

Building MOAB library with TPL dependencies needed for E3SM

These instructions assume you have already ported E3SM to your machine (you have it defined in config_machines.xml and can build and run cases)

There are several option dependencies for MOAB that are required for MOAB-in-E3SM. Features and tools get automatically built and enabled based on the dependencies when configuring the MOAB library.

  • MPI: MOAB supports both MPICH- and OpenMPI-based libraries configured externally to enable scalable mesh manipulation algorithms. Needed for MOAB-coupler.

  • HDF5: To manage the data dependencies and to natively support parallel I/O, MOAB uses a native HDF5-based file format to represent the entire MOAB data model. If MPI is enabled, HDF5 should be built with parallel support. This is required because TempestRemap (below) needs it. It also allows you to output MOAB h5m files which are useful in debugging.

  • NetCDF: MOAB library optionally depends on the NetCDF libraries (C) to compile the ExodusII reader/writer. If MPI is enabled, then the underlying HDF5 dependency for NetCDF should have parallel support i.e., serial HDF5 dependency will not pass configuration checks. If you built it for E3SM, use that one.

  • Parallel-NetCDF: MOAB library optionally depends on the parallel-NetCDF (pnetcdf) libraries to compile the support for parallel reader and writer of NetCDF meshes and linear map weight files. This is a required dependency to enable parallel I/O of MPAS, SCRIP, ESMF, and Exodus meshes. Use the same one you built for E3SM.

  • Zoltan: Support for online partitioning through Zoltan (and its dependencies on Scotch, ParMetis etc) can be utilized through the partitioner tool and at runtime with appropriate file read options (PARTITION_METHOD=RCBZOLTAN;). This dynamic rebalancing is important to achieve good scalability when reading nc files. If reading parallel h5m files, we recommend using the offline partitioning tool with mbpart with -z RCB to use the Recursive Bisection geometric partitioner implementation in Zoltan. This is required for E3SM’s moab coupler.

  • TempestRemap: Provides support for online remapping of Climate field data on unstructured spherical meshes. Required.

  • Eigen3: A substitute for BLAS/LAPACK interfaces. Required with TempestRemap as we store the SparseMatrix internally on every task using the Eigen3 data-structures. Required.

Optional dependencies (not needed to run the default moab coupler)

  • Metis/ParMetis: MOAB can optionally use the Metis or ParMetis library for partitioning mesh files in an offline form through the mbpart tool. This may be useful to pre-partition the mesh apriori in offline remapping workflows.

Configuring and building MOAB from source

First, clone the sources from the repository using the command

git clone https://bitbucket.org/fathomteam/moab.git

Currently, both CMake and Autotools are maintained simultaneously to support all platforms (including Windows). In this section, we will focus only on the autotools configuration system as this remains the most tested configuration approach for now. In the future (FY24Q3), we will deprecate the autotools configuration system and use the CMake-based approach instead. When this happens, the instructions will be updated accordingly.

For the autotools-based configuration, we recommend a minimum autoconf version of 2.69.

Autotools configuration

  • Run autoreconf -fi to generate the configure script

  • Run ./configure --help script in the MOAB source directory to see the list of configure options

  • Use --prefix=INSTALL_DIR to specify installation directory for MOAB

(Below use the same compiler and mpi and I/O libraries you have set up for E3SM. If you have mutiple compiler-mpi options, you would need to build MOAB for each.)

  • Override default compilers with environment variables or configure-time options with the compilers you use for E3SM: CC, CXX, FC, F77

  • For the MPI installed for E3SM, use --with-mpi=$MPI_DIR

  • For the HDF5, NetCDF, PNetCDF installed, use --with-hdf5=$HDF5_DIR --with-netcdf=$NETCDF_DIR --with-pnetcdf=$PNETCDF_DIR to specify these external dependencies.

  • Zoltan must currently be installed separately. use --with-zoltan=$ZOLTAN_DIR

  • For TempestRemap, if you pre-installed it, then use --with-tempestremap=$TEMPESTREMAP_DIR during configuration, or you can use the auto-download option to let MOAB automatically build the dependency with the configure option --download-tempestremap(to download the latest release) OR --download-tempestremap=master (to build from TempestRemap git repository, master branch).

  • Now execute the configure script with desired configuration options either in-source or out-of-source (recommended) directory. A typical MOAB configuration command with all TPLs enabled for E3SM might look like the one shown below, where $PACKAGE_DIR indicates the location of the installation prefix for the particular TPL.

    $MOAB_SRC/configure CC=mpicc CXX=mpicxx FC=mpif90 F77=mpif77 \
                    --prefix=$MOAB_INSTALL_DIR \
                    --enable-debug --enable-optimize \
                    --with-mpi=$MPI_DIR \
                    --with-eigen3=$EIGEN3_INCLUDE_DIR \
                    --with-hdf5=$HDF5_DIR \
                    --with-netcdf=$NETCDF_DIR \
                    --with-pnetcdf=$PNETCDF_DIR \
                    --with-tempestremap=$TEMPESTREMAP_DIR \
                    --with-zoltan=$ZOLTAN_DIR

Once configuration with the autotools workflow is complete in the build directory, run the following commands to build the library and verify the build.

  • Compile MOAB and supported tools: make -j4

  • Verify configuration and build setup: make check

Next to install the compiled libraries, headers, and tools, run: make install

If you experience any issues during configuration or linking phases of MOAB, please email the mailing list: moab-dev@mcs.anl.gov , or contact Vijay M , Iulian Grindeanu or #coupler on Slack for additional help.

Once you have verified MOAB has been built correctly, you need to tell the CIME machine config that it is available.

Pre-built recipes (Conda)

MOAB comes pre-built as part of a conda package to use offline tools like mbtempest. DO NOT use the conda installation for MOAB for the MOAB coupler in E3SM as often the preinstalled package will be suboptimal for the cluster or machine in use. However, such installations are useful for offline remapping (mbtempest) and computational analysis workflows (with Python interfaces) using MOAB (and pyMOAB).

Once you have a conda environment setup, use the command below to install MOAB with TempestRemap enabled (with or without MPI).

With MPI: conda install -c conda-forge moab=*=mpi_mpich_tempest*

Without MPI: conda install -c conda-forge moab=*=nompi_tempest*

This method is useful for quickly getting started with MOAB and TempestRemap interfaces, as the build is consistent with all third-party libraries (TPLs) available prebuilt with MPI enabled or disabled. For more extensive TPL support in Conda, use the E3SM-Unified conda package.

  • No labels