O_21_LI Third Party Support for MPAS-LI within ACME Design Document

Owned by Tanya Reshel
Last updated: Oct 15, 2015 by Andy SalingerVersion comment
8 min read

The Design Document page provides a description of the algorithms, implementation and planned testing including unit, verification, validation and performance testing. Please read  Step 1.3 Performance Expectations that explains feature documentation requirements from the performance group point of view. 

Design Document

 

 Click here for instructions to fill up the table below ......

The first table in Design Document gives overview of this document, from this info the Design Documents Overview page is automatically created.

In the table below, 4.Equ means Equations and Algorithms, 5.Ver means Verification, 6.Perf - Performance, 7. Val - Validation

  • Equations: Document the equations that are being solved and describe algorithms
  • Verification Plans: Define tests that will be run to show that implementation is correct and robust. Involve unit tests to cover range of inputs as well as benchmarks.
  • Performance expectations: Explain the expected performance impact from this development
  • Validation Plans: Document what process-based, stand-alone component, and coupled model runs will be performed, and with what metrics will be used to assess validity

Use the symbols below (copy and paste) to indicate if the section is in progress or done or not started.

In the table below, 4.Equ means Equations and Algorithms, 5.Ver means Verification, 6.Perf - Performance, 7. Val - Validation,   (tick) - completed, (warning) - in progress, (error) - not done

 

Overview table for the owner and an approver of this feature

1.Description

LI Third Party Support for MPAS-LI within ACME
2.OwnerAndy Salinger
3.Created 
4.EquN/A (build system modifications)
5.Ver(tick)
6.Perf(tick)
7.Val(tick)
8.Approver 
9.Approved Date 
V1.0Accepted

 

 Click here for Table of Contents ...

Table of Contents

 

 

 


Title: O_21_LI Third Party Support for MPAS-LI within ACME

Requirements and Design

ACME Ocean and Ice Group

Date:  9-4-2015

Summary

ACME v1 requires that MPASLI build and run with the First-order solver written in the Albany framework.  Albany is a standalone library written in C++ and based on Trilinos.  In standalone MPAS, an Albany build is enabled by passing "ALBANY=true" on the build line and defining the needed libraries and their locations with the MPAS_EXTERNAL_LIBS shell variable.

Requirements


Requirement: ACME builds and runs with Albany linked in

Date last modified: 2015/9/4
Contributors: Matt Hoffman, Andy Salinger, Mauro Perego, others?

The model needs to build and run and not be overly cumbersome to use and maintain.  MPASLI configurations need to exist with and without Albany.

 

Algorithmic Formulations


There are no algorithmic formulations for this task as it pertains only to build system modifications.

Design and Implementation

Implementation: ACME builds and runs with Albany linked in

Date last modified: 2015/9/4

Contributors: Matt Hoffman, Andy Salinger, Mauro Perego, others?

 

The basic approach is to duplicate how Trilinos was added to CESM for use with CISM.  Trilinos was added to CESM/ACME in a general way that potentially allows any component to use it.  Most of the logic is in the highest level Makefile which allows the ACME executable to have the necessary libraries linked in.  You can get a sense of what is set up for Trilinos by doing 'git grep -i TRILINOS' from the scripts directory:https://gist.github.com/matthewhoffman/fc46649be672649197fc

Specifically, there are a number of pieces required:


  1. Use multiple compsets/component names to enable MPASLI with and without Albany.

To enable builds of MPASLI that do and do not include Albany, compsets for each configuration will be created.  There will be different component names for MPASLI with and without Albany (e.g., MPASLISIA, MPASLIALBFO), similar to the different versions of CISM that are supported (CISM1, CISM2S, CISM2P, as defined in config_compsets.xml). 

 

2. Use logic and variables in config_compset.xml to indicate needed adjustments to build and namelist generation

Following CISM Trilinos implementation, we will define MPASLI_USE_ALBANY and use it in config_compset.xml:
<MPASLI_USE_ALBANY compset="_MPASLISIA">FALSE</MPASLI_USE_ALBANY>
<MPASLI_USE_ALBANY compset="_MPASLIALB">TRUE</MPASLI_USE_ALBANY>
 
Component usage:
MPASLI_USE_ALBANY is checked by components/mpasli/bld/mpasli.buildlib to trigger an Albany-enabled configuration of MPASLI.
That variable will also need to be used to adjust the namelist as needed.
ACME-wide usage:
MPASLI_USE_ALBANY is checked by the script cime/scripts/Tools/cesm_build.pl and then an CIME-wide variable USE_ALBANY is activated.  USE_ALBANY is then checked in the root Makefile to enable linking to Albany when the acme executable is built.
3. Include the list of necessary Albany libs for linking into the executable
Note MPAS_EXTERNAL_LIBS is not needed in building the MPASLI component since ACME is doing the linking!  However, that means that ACME needs to have that list of libs specified at link time.
In standalone MPAS, MPAS_EXTERNAL_LIBS is the variable that indicates the libraries and paths for the Albany and Trilinos (and a few other things like MPI, boost, etc.) libraries.  Right now on a generic linux machine it looks something like this:

echo $MPAS_EXTERNAL_LIBS
-L/home/mhoffman/software/albany/albany-install-mpas/lib -lmpasInterface -lalbanyLib -lalbanySTK -lFELIX -lalbanyAdapt -lalbanySTKRebalance -L/home/mhoffman/software/trilinos/trilinos-git-install/lib -lpiro -lstokhos_muelu -lstokhos_ifpack2 -lstokhos_amesos2 -lstokhos_tpetra -lstokhos_sacado -lstokhos -lrythmos -lmuelu-adapters -lmuelu-interface -lmuelu -llocathyra -llocaepetra -llocalapack -lloca -lnoxepetra -lnoxlapack -lnox -lphalanx -lstk_unit_test_utils -lstk_search -lstk_io_util -lstk_io -lstk_mesh_base -lstk_topology -lstk_util_use_cases -lstk_util_registry -lstk_util_parallel -lstk_util_diag -lstk_util_env -lstk_util_util -lstk_unit_test_utils -lstk_search -lstk_io_util -lstk_io -lstk_mesh_base -lstk_topology -lstk_util_use_cases -lstk_util_registry -lstk_util_parallel -lstk_util_diag -lstk_util_env -lstk_util_util -lintrepid -lteko -lstratimikos -lstratimikosbelos -lstratimikosaztecoo -lstratimikosamesos -lstratimikosml -lstratimikosifpack -lifpack2-adapters -lifpack2 -lzoltan2 -lanasazitpetra -lModeLaplace -lanasaziepetra -lanasazi -lbelostpetra -lbelosepetra -lbelos -lml -lifpack -lmapvarlib -lfastqlib -lblotlib -lplt -lsvdi_cgi -lsvdi_cdr -lsuplib -lsupes -laprepro_lib -lchaco -lIonit -lIotr -lIohb -lIogn -lIopg -lIoexo_fac -lIopx -lIofx -lIoex -lIoss -lnemesis -lexodus_for -lexodus -lmapvarlib -lfastqlib -lblotlib -lplt -lsvdi_cgi -lsvdi_cdr -lsuplib -lsupes -laprepro_lib -lchaco -lIonit -lIotr -lIohb -lIogn -lIopg -lIoexo_fac -lIopx -lIofx -lIoex -lIoss -lnemesis -lexodus_for -lexodus -lpamgen_extras -lpamgen -lamesos2 -lamesos -lgaleri-xpetra -lgaleri -laztecoo -lisorropia -loptipack -lthyratpetra -lthyraepetraext -lthyraepetra -lthyracore -lthyratpetra -lthyraepetraext -lthyraepetra -lthyracore -lxpetra-sup -lxpetra-ext -lxpetra -lepetraext -ltpetraext -ltpetrainout -ltpetra -lkokkostsqr -ltpetrakernels -ltpetraclassiclinalg -ltpetraclassicnodeapi -ltpetraclassic -ltpetraext -ltpetrainout -ltpetra -lkokkostsqr -ltpetrakernels -ltpetraclassiclinalg -ltpetraclassicnodeapi -ltpetraclassic -ltriutils -lglobipack -lshards -lzoltan -lepetra -lsacado -lrtop -lteuchoskokkoscomm -lteuchoskokkoscompat -lteuchosremainder -lteuchosnumerics -lteuchoscomm -lteuchosparameterlist -lteuchoscore -lteuchoskokkoscomm -lteuchoskokkoscompat -lteuchosremainder -lteuchosnumerics -lteuchoscomm -lteuchosparameterlist -lteuchoscore -lkokkosalgorithms -lkokkoscontainers -lkokkoscore -lkokkosalgorithms -lkokkoscontainers -lkokkoscore -ltpi -lgtest /usr/lib/x86_64-linux-gnu/libX11.so /usr/lib/x86_64-linux-gnu/libboost_program_options.so /usr/lib/x86_64-linux-gnu/libboost_system.so /home/mhoffman/software/netcdf/netcdf-4.3.2-install/lib/libnetcdf.so /usr/lib/x86_64-linux-gnu/libhdf5.so /usr/lib/x86_64-linux-gnu/libz.so /usr/lib/liblapack.so /usr/lib/libblas.so -L/usr/lib -lmpi_cxx -lmpi_f90 -lstdc++

 

it is constructed like this:

export MPAS_EXTERNAL_LIBS="$ALBANY_LIBS $Trilinos_LIBRARY_DIRS $Trilinos_LIBRARIES $Trilinos_TPL_LIBRARIES -L/usr/lib -lmpi_cxx -lmpi_f90 -lstdc++"

where ALBANY_LIBS is manually-maintained based on Albany development changes, but currently looks like:

export ALBANY_LIBS="-L/home/mhoffman/software/albany/albany-install-mpas/lib -lmpasInterface -lalbanyLib -lalbanySTK -lFELIX -lalbanyAdapt -lalbanySTKRebalance"

and the Trilinos-related libraries are defined in the "Makefile.export.Trilinos_install" file created when building Trilinos.  Currently it looks something like this:
## The project library directories.
export Trilinos_LIBRARY_DIRS="-L/home/mhoffman/software/trilinos/trilinos-git-install/lib"
## The project libraries.
export Trilinos_LIBRARIES="-lpiro -lstokhos_muelu -lstokhos_ifpack2 -lstokhos_amesos2 -lstokhos_tpetra -lstokhos_sacado -lstokhos -lrythmos -lmuelu-adapters -lmuelu-interface -lmuelu -llocathyra -llocaepetra -llocalapack -lloca -lnoxepetra -lnoxlapack -lnox -lphalanx -lstk_unit_test_utils -lstk_search -lstk_io_util -lstk_io -lstk_mesh_base -lstk_topology -lstk_util_use_cases -lstk_util_registry -lstk_util_parallel -lstk_util_diag -lstk_util_env -lstk_util_util -lstk_unit_test_utils -lstk_search -lstk_io_util -lstk_io -lstk_mesh_base -lstk_topology -lstk_util_use_cases -lstk_util_registry -lstk_util_parallel -lstk_util_diag -lstk_util_env -lstk_util_util -lintrepid -lteko -lstratimikos -lstratimikosbelos -lstratimikosaztecoo -lstratimikosamesos -lstratimikosml -lstratimikosifpack -lifpack2-adapters -lifpack2 -lzoltan2 -lanasazitpetra -lModeLaplace -lanasaziepetra -lanasazi -lbelostpetra -lbelosepetra -lbelos -lml -lifpack -lmapvarlib -lfastqlib -lblotlib -lplt -lsvdi_cgi -lsvdi_cdr -lsuplib -lsupes -laprepro_lib -lchaco -lIonit -lIotr -lIohb -lIogn -lIopg -lIoexo_fac -lIopx -lIofx -lIoex -lIoss -lnemesis -lexodus_for -lexodus -lmapvarlib -lfastqlib -lblotlib -lplt -lsvdi_cgi -lsvdi_cdr -lsuplib -lsupes -laprepro_lib -lchaco -lIonit -lIotr -lIohb -lIogn -lIopg -lIoexo_fac -lIopx -lIofx -lIoex -lIoss -lnemesis -lexodus_for -lexodus -lpamgen_extras -lpamgen -lamesos2 -lamesos -lgaleri-xpetra -lgaleri -laztecoo -lisorropia -loptipack -lthyratpetra -lthyraepetraext -lthyraepetra -lthyracore -lthyratpetra -lthyraepetraext -lthyraepetra -lthyracore -lxpetra-sup -lxpetra-ext -lxpetra -lepetraext -ltpetraext -ltpetrainout -ltpetra -lkokkostsqr -ltpetrakernels -ltpetraclassiclinalg -ltpetraclassicnodeapi -ltpetraclassic -ltpetraext -ltpetrainout -ltpetra -lkokkostsqr -ltpetrakernels -ltpetraclassiclinalg -ltpetraclassicnodeapi -ltpetraclassic -ltriutils -lglobipack -lshards -lzoltan -lepetra -lsacado -lrtop -lteuchoskokkoscomm -lteuchoskokkoscompat -lteuchosremainder -lteuchosnumerics -lteuchoscomm -lteuchosparameterlist -lteuchoscore -lteuchoskokkoscomm -lteuchoskokkoscompat -lteuchosremainder -lteuchosnumerics -lteuchoscomm -lteuchosparameterlist -lteuchoscore -lkokkosalgorithms -lkokkoscontainers -lkokkoscore -lkokkosalgorithms -lkokkoscontainers -lkokkoscore -ltpi -lgtest"
## The project tpl libraries
export Trilinos_TPL_LIBRARIES="/usr/lib/x86_64-linux-gnu/libX11.so /usr/lib/x86_64-linux-gnu/libboost_program_options.so /usr/lib/x86_64-linux-gnu/libboost_system.so /home/mhoffman/software/netcdf/netcdf-4.3.2-install/lib/libnetcdf.so /usr/lib/x86_64-linux-gnu/libhdf5.so /usr/lib/x86_64-linux-gnu/libz.so /usr/lib/liblapack.so /usr/lib/libblas.so"

In standalone MPASLI, we typically have created environment variables to store this information, so that within a user's environment they simply need to compile MPAS with, e.g., "make gfortran ALBANY=true" and the MPAS_EXTERNAL_LIBS environment variable is automatically included in the build.  However, in ACME it appears that a user's environment is ignored when building ACME, so this approach is not possible. 

Andy Salinger has recently added files to the Albany build that automatically construct this list of libs and paths.  Because this list is in a standard place (the root of the Albany install), ACME can read this list and add it to the link line in the ACME root Makefile.


4. Setup Albany information for each machine
The above changes still require that a path to Albany is specified for each machine. Each machine needs the path to Albany specified in cime/machines-acme/config_compilers.xml.  If modules are to be loaded, that should happen in cime/machines-acme/env_mach_specific.MACH.  Note that the way things are implemented above, even if the module loads ALBANY_LINK_LIBS or MPAS_EXTERNAL_LIBS, ACME is not using those environment variables - it is creating its own version of ALBANY_LINK_LIBS this way in the Makefile:   include $(ALBANY_PATH)/export_albany.in

Therefore, an Albany module is not required.

Planned Verification and Unit Testing 

Verification and Unit Testing: Verify Working Builds on LCF (and other testing) Machines

Date last modified: 2015/9/4

Contributors: Matt Hoffman, Andy Salinger


Verification testing will consist of:

  1. confirming that ACME can successfully build with MPAS-LI + Albany/Trilinos
  2. confirming that MPAS-LI + Albany/Trilinos can run successfully within ACME
  3. confirming that results from (2) above are consistent with stand-alone runs using MPAS-LI + Albany/Trilinos

Status on Edison:

As of 9/22 we have a successful build of acme/mpasli/albany using the Intel compilers. The pre-existing ACME infrastructure for linking to Trilinos was replicated for Albany. The Albany code was modified to export an environment variable with the complete link line needed to access AlbanyTrilinos, which is then read into the ACME build system. We installed AlbanyTrilinos, compiled with Intel, in the acme project space. There were issues with regard to incompatible netcdf/hdf5 versions between ACME and Albany, and the wrong selection of the C++ compiler in a Makefile, that were the last steps to be sorted out. There is still some clean-up and documentation needed to finish this off.

Status on Titan:

9/25 update: I was successful in linking the acme code to AlbanyTrilinos on Titan. I used GNU for AlbanyTrilinos and PGI for ACME. I had to hand-edit the ALBANY_LINK_LIBS libraries that Albany exports to remove several references to libraries that were not found, and add a specific libstdc++.a that was not the one found by the linker. For both of those modifications, it seems to be an issue of GNU version. PGI seems to provide paths to gcc4.3 libraries while the default gcc compiler is gcc4.8. 

All of this is in my home directory, including inputdata.

Status on Mira:

9/23 update: The XLF compiler suite on Mira does not compile C++11 code in AlbanyTrilinos. We have communicated with ALCF user support, and they claim it is possible to link XLF-compiled Fortran to bgclang++-compiled C++ code. We will attempt this route.

10/15 update: I was successful in linking the acme executable with XLF for fortran and bgclang++ for C++. In the end, I used the C++ compiler to link and had to add several fortran libraries to the link line. The name mangling was different then all other compiler compilations, so had a temporary hack of adding several underscores in a source file. Working on a portable solution. 


Status on LANL HPC:

Albany and Trilinos modules have been created on Mustang and Wolf.  The "MPASLIALB_ONLY" test case builds and runs to completion.  History file output is BFB with the run same configuration in standalone MPAS.  This required the addition of commit 6beb280 (cherry-picked and modified from matthewhoffman/mpasli/lnd_glc_coupling branch) to get the diagnostic solve of velocity occuring on initialization so that evolution on the first time step was correct.


Planned Validation Testing 


Validation testing is not applicable for this task. Verification testing - i.e. that the changes to the build system needed for MPAS-LI in ACME to call the third-party libraries - should be adequate.

Planned Performance Testing


Performance Testing:

Date last modified: 2015/9/22
Contributors: Stephen Price

 

This is part of a new land ice model component so there are no previous benchmarks for model performance testing. Performance testing might consist of running ACME in two similar model configurations, one with an inactive (data) land ice model component and one with an active land ice model component (for which MPAS-LI + Albany/Trilinos are actively calculating ice sheet evolution as part of the coupled climate simulation).