Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 6 Next »

As part of the efforts in the CMDV project, interfaces to integrate the MOAB unstructured mesh library with the TempestRemap remapping tool have been undertaken. This work has led to the development of a new offline remapping tool called mbtempest, which exposes the functionality to compute the supermesh or intersection mesh between two unstructured source and target component grids, in addition to using this supermesh for computing the remapping weights to project solutions between the grids. This functionality is part of the critical worflow with E3SM, where the generated remapping weights in the offline step are consumed by MCT at runtime to seamlessly transfer solution data between components (atm↔ocn, atm↔lnd, etc).

Getting started

In order to build the MOAB-TempestRemap stack with parallel MPI launch support, we suggest the following list of commands. First define an installation prefix directory where the stack of library, includes and tools will be installed. Let us call this as the $INSTALL_PREFIX environment variable.

Dependencies

Before getting started, for your architecture of choice, whether that is your laptop or a LCF machine, create a list of following compatible environment variables that can be used to build the stack.

  1. MPI-enabled C, C++, and Fortran compiler wrappers that are exported in the local environment as $CC, $CXX, and $FC.
  2. Next, verify installations of dependent libraries such as $HDF5_DIR and $NETCDF_DIR that have been compiled with MPI support using the $CC, $CXX, $FC compilers.
  3. Get Eigen3 package from the webpage and untar to the $INSTALL_PREFIX/eigen3 directory with the following command
    a. Download: wget https://bitbucket.org/eigen/eigen/get/3.3.7.tar.gz  OR  curl https://bitbucket.org/eigen/eigen/get/3.3.7.tar.gz -O
    b. Move: mv eigen-eigen* $INSTALL_PREFIX/eigen3

Build

To get the entire stack working correctly, we need to find parallel-enabled dependency installations for HDF5 and NetCDF that are built with MPI library support for the current architecture

  1. TempestRemap

    a. Clone repository: `git clone https://github.com/ClimateGlobalChange/tempestremap.git tempestremap`
    b. Create build dir: cd tempestremap && mkdir build
    c. Generate configure script: autoreconf -fi
    d. Go to build dir: cd build
    e. Configure: ../configure --prefix=$INSTALL_PREFIX/tempestremap --with-netcdf=$NETCDF_DIR --with-hdf5=$HDF5_DIR CC=$CC CXX=$CXX CXXFLAGS="-g -O2"
    f. Build and install: make all && make install

    At the end of this series of steps, the TempestRemap libraries and tools (GenerateCSMesh, GenerateICOMesh, GenerateOverlapMesh, GenerateOfflineMap among others) will be installed in $INSTALL_PREFIX/tempestremap directory.

  2. MOAB

    a. Clone repository: git clone https://bitbucket.org/fathomteam/moab.git moab
    b. Create build dir: cd moab && mkdir build
    c. Generate configure script: autoreconf -fi
    d. Go to build dir: cd build
    e. Configure: ../configure --prefix=$INSTALL_PREFIX/moab --with-mpi --with-tempestremap=$INSTALL_PREFIX/tempestremap --with-netcdf=$NETCDF_DIR --with-hdf5=$HDF5_DIR CC=$CC FC=$FC F77=$FC CXX=$CXX CXXFLAGS="-g -O2" --with-eigen3=$INSTALL_PREFIX/eigen3
    f. Build and install: make all && make install

    If steps (a)-(f) pass successfully, the MOAB libraries and tools, along with interfaces for TempestRemap will be installed in $INSTALL_PREFIX/moab directory. The offline remapping weight computation tool, mbtempest, will also be installed during this process and can then be used standalone to generate the weight files as needed.

What is mbtempest ?

The mbtempest tool exposes the algorithms and interfaces to invoke TempestRemap through MOAB to generate remapping weights for combinations of discretizations and unstructured meshes defined on a source grid to a target grid. Most of the options supported by TempestRemap tools are provided in this unified interface with one key difference: the entire workflow makes use of MPI parallelism. This implies that the overlap mesh computation that would normally use the GenerateOverlapMesh tool in TempestRemap would be replaced by the MOAB's parallel implementation of an advancing front intersection computation. This intersection mesh can subsequently be written out to file as an intermediate step or used by mbtempest to generate the remapping weights needed to project a solution field from source to target component grid. Prescriptions to preserve conservation of scalar/flux data and to impose monotonicity constraints are also available as options to pass to the mbtempest tool.

[bash]> tools/mbtempest -h
Usage: mbtempest --help | [options] 
Options: 
  -h [--help]       : Show full help text
  -t [--type] <int> : Type of mesh (default=CS; Choose from [CS=0, RLL=1, ICO=2, OVERLAP_FILES=3, OVERLAP_MEMORY=4, OVERLAP_MOAB=5])
  -r [--res] <int>  : Resolution of the mesh (default=5)
  -d [--dual]       : Output the dual of the mesh (generally relevant only for ICO mesh)
  -w [--weights]    : Compute and output the weights using the overlap mesh (generally relevant only for OVERLAP mesh)
  -c [--noconserve] : Do not apply conservation to the resultant weights (relevant only when computing weights)
  -v [--volumetric] : Apply a volumetric projection to compute the weights (relevant only when computing weights)
  -n [--monotonic] <int>: Ensure monotonicity in the weight generation
  -l [--load] <arg> : Input mesh filenames (a source and target mesh)
  -o [--order] <int>: Discretization orders for the source and target solution fields
  -m [--method] <arg>: Discretization method for the source and target solution fields
  -g [--global_id] <arg>: Tag name that contains the global DoF IDs for source and target solution fields
  -f [--file] <arg> : Output remapping weights filename
  -i [--intx] <arg> : Output TempestRemap intersection mesh filename

The mbtempest tool can be used to generate CS (cubed-sphere), RLL (lat/lon), ICO (triangular) and polygonal (MPAS-like) meshes through appropriate invocation with the -t [--type] argument in combination with -r [--res] option. Some examples for these are provided below.

Cubed-Sphere meshes

[bash]> tools/mbtempest -t 0 -r 120 -f outputCSMesh.exo
Creating TempestRemap Mesh object ...
=========================================================
..Generating mesh with resolution [120]
..Writing mesh to file [outputCSMesh.exo] 
Nodes per element
..Block 1 (4 nodes): 86400
..Mesh generator exited successfully
=========================================================
[LOG] Time taken to create Tempest mesh: max = 0.0571213, avg = 0.0571213
[bash]> tools/mbsize outputCSMesh.exo
File outputCSMesh.exo:
   type  count   total                            minimum                            average                                rms                            maximum                           std.dev.
------- ------ ------- ---------------------------------- ---------------------------------- ---------------------------------- ---------------------------------- ----------------------------------
   Quad  86400      13                         0.00012195                         0.00014544                         0.00014602                         0.00017133                         1.3029e-05
1D Side 345600 4.2e+03                          0.0092562                            0.01218                           0.012217                            0.01309                         0.00094931
 Vertex  86402



  • No labels