Versions Compared

Old Version 7

changes.mady.by.user Stephen Price

Saved on

compared with

New Version Current

changes.mady.by.user Andy Salinger

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Added successful Mira build. Still need some work on portability and some cleanup.

The Design Document page provides a description of the algorithms, implementation and planned testing including unit, verification, validation and performance testing. Please read  Step 1.3 Performance Expectations that explains feature documentation requirements from the performance group point of view. 

Design Document

 

Expand
titleClick here for instructions to fill up the table below ......

The first table in Design Document gives overview of this document, from this info the Design Documents Overview page is automatically created.

In the table below, 4.Equ means Equations and Algorithms, 5.Ver means Verification, 6.Perf - Performance, 7. Val - Validation

  • Equations: Document the equations that are being solved and describe algorithms
  • Verification Plans: Define tests that will be run to show that implementation is correct and robust. Involve unit tests to cover range of inputs as well as benchmarks.
  • Performance expectations: Explain the expected performance impact from this development
  • Validation Plans: Document what process-based, stand-alone component, and coupled model runs will be performed, and with what metrics will be used to assess validity

Use the symbols below (copy and paste) to indicate if the section is in progress or done or not started.

In the table below, 4.Equ means Equations and Algorithms, 5.Ver means Verification, 6.Perf - Performance, 7. Val - Validation,   (tick) - completed, (warning) - in progress, (error) - not done

 

Page Properties
idFeature_PR
Info

Overview table for the owner and an approver of this feature

1.Description

O_21_LI Third Party Support for MPAS-LI within ACME
2.OwnerAndy Salinger
3.Created 
4.EquN/A (build system modifications)
5.Ver(error)(tick)
6.Perf(error)(tick)
7.Val(error)(tick)
8.Approver 
9.Approved Date 
V1.0Accepted

 

Expand
titleClick here for Table of Contents ...
Panel

Table of Contents

Table of Contents

 

 

 


Title: O_21_LI Third Party Support for MPAS-LI within ACME

Requirements and Design

ACME Ocean and Ice Group

Date:  9-4-2015

Summary

ACME v1 requires that MPASLI build and run with the First-order solver written in the Albany framework.  Albany is a standalone library written in C++ and based on Trilinos.  In standalone MPAS, an Albany build is enabled by passing "ALBANY=true" on the build line and defining the needed libraries and their locations with the MPAS_EXTERNAL_LIBS shell variable.

Requirements


Requirement: ACME builds and runs with Albany linked in

Date last modified: 2015/9/4
Contributors: Matt Hoffman, Andy Salinger, Mauro Perego, others?

The model needs to build and run and not be overly cumbersome to use and maintain.  MPASLI configurations need to exist with and without Albany.

 

Algorithmic Formulations


There are no algorithmic formulations for this task as it pertains only to build system modifications.

Design and Implementation

Implementation: ACME builds and runs with Albany linked in

Date last modified: 2015/9/4

Contributors: Matt Hoffman, Andy Salinger, Mauro Perego, others?

  1. Use multiple compsets/component names to enable MPASLI with and without Albany.

To enable builds of MPASLI that do and do not include Albany, compsets for each configuration will be created.  There will be different component names for MPASLI with and without Albany (e.g., MPASLISIA, MPASLIALBFO), similar to the different versions of CISM that are supported (CISM1, CISM2S, CISM2P, as defined in config_compsets.xml). 

 

2. Use *_CONFIG_OPTS variable to indicate needed adjustments to build and namelist generation

The *_CONFIG_OPTS that is automatically parsed for each component will be used to enable the HO physics with Albany:

<MPASLI_CONFIG_OPTS compset="_MPASLIALBFO">-phys fo</MPASLI_CONFIG_OPTS>

The MPASLI_CONFIG_OPTS variable can then be used at build time in models/glc/mpasli/bld/mpasli.buildexe.csh to add the "ALBANY=true" compile flag for MPAS.
This can also be used to adjust the namelist options to select the Albany velocity solver for these compsets.
3. Set MPAS_EXTERNAL_LIBS shell variable and give the build system access to it
MPAS_EXTERNAL_LIBS is the variable that indicates the libraries and paths for the Albany and Trilinos (and a few other things like MPI, boost, etc.) libraries.  Right now on a generic linux machine it looks something like this:
echo $MPAS_EXTERNAL_LIBS
-L/home/mhoffman/software/albany/albany-install-mpas/lib -lmpasInterface -lalbanyLib -lalbanySTK -lFELIX -lalbanyAdapt -lalbanySTKRebalance -L/home/mhoffman/software/trilinos/trilinos-git-install/lib -lpiro -lstokhos_muelu -lstokhos_ifpack2 -lstokhos_amesos2 -lstokhos_tpetra -lstokhos_sacado -lstokhos -lrythmos -lmuelu-adapters -lmuelu-interface -lmuelu -llocathyra -llocaepetra -llocalapack -lloca -lnoxepetra -lnoxlapack -lnox -lphalanx -lstk_unit_test_utils -lstk_search -lstk_io_util -lstk_io -lstk_mesh_base -lstk_topology -lstk_util_use_cases -lstk_util_registry -lstk_util_parallel -lstk_util_diag -lstk_util_env -lstk_util_util -lstk_unit_test_utils -lstk_search -lstk_io_util -lstk_io -lstk_mesh_base -lstk_topology -lstk_util_use_cases -lstk_util_registry -lstk_util_parallel -lstk_util_diag -lstk_util_env -lstk_util_util -lintrepid -lteko -lstratimikos -lstratimikosbelos -lstratimikosaztecoo -lstratimikosamesos -lstratimikosml -lstratimikosifpack -lifpack2-adapters -lifpack2 -lzoltan2 -lanasazitpetra -lModeLaplace -lanasaziepetra -lanasazi -lbelostpetra -lbelosepetra -lbelos -lml -lifpack -lmapvarlib -lfastqlib -lblotlib -lplt -lsvdi_cgi -lsvdi_cdr -lsuplib -lsupes -laprepro_lib -lchaco -lIonit -lIotr -lIohb -lIogn -lIopg -lIoexo_fac -lIopx -lIofx -lIoex -lIoss -lnemesis -lexodus_for -lexodus

 

The basic approach is to duplicate how Trilinos was added to CESM for use with CISM.  Trilinos was added to CESM/ACME in a general way that potentially allows any component to use it.  Most of the logic is in the highest level Makefile which allows the ACME executable to have the necessary libraries linked in.  You can get a sense of what is set up for Trilinos by doing 'git grep -i TRILINOS' from the scripts directory:https://gist.github.com/matthewhoffman/fc46649be672649197fc

Specifically, there are a number of pieces required:


  1. Use multiple compsets/component names to enable MPASLI with and without Albany.

To enable builds of MPASLI that do and do not include Albany, compsets for each configuration will be created.  There will be different component names for MPASLI with and without Albany (e.g., MPASLISIA, MPASLIALBFO), similar to the different versions of CISM that are supported (CISM1, CISM2S, CISM2P, as defined in config_compsets.xml). 

 

2. Use logic and variables in config_compset.xml to indicate needed adjustments to build and namelist generation

Following CISM Trilinos implementation, we will define MPASLI_USE_ALBANY and use it in config_compset.xml:
<MPASLI_USE_ALBANY compset="_MPASLISIA">FALSE</MPASLI_USE_ALBANY>
<MPASLI_USE_ALBANY compset="_MPASLIALB">TRUE</MPASLI_USE_ALBANY>
 
Component usage:
MPASLI_USE_ALBANY is checked by components/mpasli/bld/mpasli.buildlib to trigger an Albany-enabled configuration of MPASLI.
That variable will also need to be used to adjust the namelist as needed.
ACME-wide usage:
MPASLI_USE_ALBANY is checked by the script cime/scripts/Tools/cesm_build.pl and then an CIME-wide variable USE_ALBANY is activated.  USE_ALBANY is then checked in the root Makefile to enable linking to Albany when the acme executable is built.
3. Include the list of necessary Albany libs for linking into the executable
Note MPAS_EXTERNAL_LIBS is not needed in building the MPASLI component since ACME is doing the linking!  However, that means that ACME needs to have that list of libs specified at link time.
In standalone MPAS, MPAS_EXTERNAL_LIBS is the variable that indicates the libraries and paths for the Albany and Trilinos (and a few other things like MPI, boost, etc.) libraries.  Right now on a generic linux machine it looks something like this:

echo $MPAS_EXTERNAL_LIBS
-L/home/mhoffman/software/albany/albany-install-mpas/lib -lmpasInterface -lalbanyLib -lalbanySTK -lFELIX -lalbanyAdapt -lalbanySTKRebalance -L/home/mhoffman/software/trilinos/trilinos-git-install/lib -lpiro -lstokhos_muelu -lstokhos_ifpack2 -lstokhos_amesos2 -lstokhos_tpetra -lstokhos_sacado -lstokhos -lrythmos -lmuelu-adapters -lmuelu-interface -lmuelu -llocathyra -llocaepetra -llocalapack -lloca -lnoxepetra -lnoxlapack -lnox -lphalanx -lstk_unit_test_utils -lstk_search -lstk_io_util -lstk_io -lstk_mesh_base -lstk_topology -lstk_util_use_cases -lstk_util_registry -lstk_util_parallel -lstk_util_diag -lstk_util_env -lstk_util_util -lstk_unit_test_utils -lstk_search -lstk_io_util -lstk_io -lstk_mesh_base -lstk_topology -lstk_util_use_cases -lstk_util_registry -lstk_util_parallel -lstk_util_diag -lstk_util_env -lstk_util_util -lintrepid -lteko -lstratimikos -lstratimikosbelos -lstratimikosaztecoo -lstratimikosamesos -lstratimikosml -lstratimikosifpack -lifpack2-adapters -lifpack2 -lzoltan2 -lanasazitpetra -lModeLaplace -lanasaziepetra -lanasazi -lbelostpetra -lbelosepetra -lbelos -lml -lifpack -lmapvarlib -lfastqlib -lblotlib -lplt -lsvdi_cgi -lsvdi_cdr -lsuplib -lsupes -laprepro_lib -lchaco -lIonit -lIotr -lIohb -lIogn -lIopg -lIoexo_fac -lIopx -lIofx -lIoex -lIoss -lnemesis -lexodus_for -lexodus -

lpamgen_extras -lpamgen -

lmapvarlib -lfastqlib -lblotlib -lplt -lsvdi_cgi -lsvdi_cdr -lsuplib -lsupes -laprepro_lib -lchaco -lIonit -lIotr -lIohb -lIogn -lIopg -lIoexo_fac -lIopx -lIofx -lIoex -lIoss -lnemesis -lexodus_for -lexodus -lpamgen_extras -lpamgen -lamesos2 -lamesos -lgaleri-xpetra -lgaleri -laztecoo -lisorropia -loptipack -lthyratpetra -lthyraepetraext -lthyraepetra -lthyracore -lthyratpetra -lthyraepetraext -lthyraepetra -lthyracore -lxpetra-sup -lxpetra-ext -lxpetra -lepetraext -ltpetraext -ltpetrainout -ltpetra -lkokkostsqr -ltpetrakernels -ltpetraclassiclinalg -ltpetraclassicnodeapi -ltpetraclassic -ltpetraext -ltpetrainout -ltpetra -lkokkostsqr -ltpetrakernels -ltpetraclassiclinalg -ltpetraclassicnodeapi -ltpetraclassic -ltriutils -lglobipack -lshards -lzoltan -lepetra -lsacado -lrtop -lteuchoskokkoscomm -lteuchoskokkoscompat -lteuchosremainder -lteuchosnumerics -lteuchoscomm -lteuchosparameterlist -lteuchoscore -lteuchoskokkoscomm -lteuchoskokkoscompat -lteuchosremainder -lteuchosnumerics -lteuchoscomm -lteuchosparameterlist -lteuchoscore -lkokkosalgorithms -lkokkoscontainers -lkokkoscore -lkokkosalgorithms -lkokkoscontainers -lkokkoscore -ltpi -lgtest /usr/lib/x86_64-linux-gnu/libX11.so /usr/lib/x86_64-linux-gnu/libboost_program_options.so /usr/lib/x86_64-linux-gnu/libboost_system.so /home/mhoffman/software/netcdf/netcdf-4.3.2-install/lib/libnetcdf.so /usr/lib/x86_64-linux-gnu/libhdf5.so /usr/lib/x86_64-linux-gnu/libz.so /usr/lib/liblapack.so /usr/lib/libblas.so -L/usr/lib -lmpi_cxx -lmpi_f90 -lstdc++

 

it is built constructed like this:

export MPAS_EXTERNAL_LIBS="$ALBANY_LIBS $Trilinos_LIBRARY_DIRS $Trilinos_LIBRARIES $Trilinos_TPL_LIBRARIES -L/usr/lib -lmpi_cxx -lmpi_f90 -lstdc++"

where ALBANY_LIBS is manually-maintained based on Albany development changes, but currently looks like:

export ALBANY_LIBS="-L/home/mhoffman/software/albany/albany-install-mpas/lib -lmpasInterface -lalbanyLib -lalbanySTK -lFELIX -lalbanyAdapt -lalbanySTKRebalance"

and the Trilinos-related libraries are defined in the "Makefile.export.Trilinos_install" file created when building Trilinos.  Currently it looks something like this:
## The project library directories.
export Trilinos_LIBRARY_DIRS="-L/home/mhoffman/software/trilinos/trilinos-git-install/lib"
## The project libraries.
export Trilinos_LIBRARIES="-lpiro -lstokhos_muelu -lstokhos_ifpack2 -lstokhos_amesos2 -lstokhos_tpetra -lstokhos_sacado -lstokhos -lrythmos -lmuelu-adapters -lmuelu-interface -lmuelu -llocathyra -llocaepetra -llocalapack -lloca -lnoxepetra -lnoxlapack -lnox -lphalanx -lstk_unit_test_utils -lstk_search -lstk_io_util -lstk_io -lstk_mesh_base -lstk_topology -lstk_util_use_cases -lstk_util_registry -lstk_util_parallel -lstk_util_diag -lstk_util_env -lstk_util_util -lstk_unit_test_utils -lstk_search -lstk_io_util -lstk_io -lstk_mesh_base -lstk_topology -lstk_util_use_cases -lstk_util_registry -lstk_util_parallel -lstk_util_diag -lstk_util_env -lstk_util_util -lintrepid -lteko -lstratimikos -lstratimikosbelos -lstratimikosaztecoo -lstratimikosamesos -lstratimikosml -lstratimikosifpack -lifpack2-adapters -lifpack2 -lzoltan2 -lanasazitpetra -lModeLaplace -lanasaziepetra -lanasazi -lbelostpetra -lbelosepetra -lbelos -lml -lifpack -lmapvarlib -lfastqlib -lblotlib -lplt -lsvdi_cgi -lsvdi_cdr -lsuplib -lsupes -laprepro_lib -lchaco -lIonit -lIotr -lIohb -lIogn -lIopg -lIoexo_fac -lIopx -lIofx -lIoex -lIoss -lnemesis -lexodus_for -lexodus -lmapvarlib -lfastqlib -lblotlib -lplt -lsvdi_cgi -lsvdi_cdr -lsuplib -lsupes -laprepro_lib -lchaco -lIonit -lIotr -lIohb -lIogn -lIopg -lIoexo_fac -lIopx -lIofx -lIoex -lIoss -lnemesis -lexodus_for -lexodus -lpamgen_extras -lpamgen -lamesos2 -lamesos -lgaleri-xpetra -lgaleri -laztecoo -lisorropia -loptipack -lthyratpetra -lthyraepetraext -lthyraepetra -lthyracore -lthyratpetra -lthyraepetraext -lthyraepetra -lthyracore -lxpetra-sup -lxpetra-ext -lxpetra -lepetraext -ltpetraext -ltpetrainout -ltpetra -lkokkostsqr -ltpetrakernels -ltpetraclassiclinalg -ltpetraclassicnodeapi -ltpetraclassic -ltpetraext -ltpetrainout -ltpetra -lkokkostsqr -ltpetrakernels -ltpetraclassiclinalg -ltpetraclassicnodeapi -ltpetraclassic -ltriutils -lglobipack -lshards -lzoltan -lepetra -lsacado -lrtop -lteuchoskokkoscomm -lteuchoskokkoscompat -lteuchosremainder -lteuchosnumerics -lteuchoscomm -lteuchosparameterlist -lteuchoscore -lteuchoskokkoscomm -lteuchoskokkoscompat -lteuchosremainder -lteuchosnumerics -lteuchoscomm -lteuchosparameterlist -lteuchoscore -lkokkosalgorithms -lkokkoscontainers -lkokkoscore -lkokkosalgorithms -lkokkoscontainers -lkokkoscore -ltpi -lgtest"
## The project tpl libraries
export Trilinos_TPL_LIBRARIES="/usr/lib/x86_64-linux-gnu/libX11.so /usr/lib/x86_64-linux-gnu/libboost_program_options.so /usr/lib/x86_64-linux-gnu/libboost_system.so /home/mhoffman/software/netcdf/netcdf-4.3.2-install/lib/libnetcdf.so /usr/lib/x86_64-linux-gnu/libhdf5.so /usr/lib/x86_64-linux-gnu/lib/libnetcdf.so /usr/lib/x86_64-linux-gnu/libhdf5.so /usr/lib/x86_64-linux-gnu/libz.so /usr/lib/liblapack.so /usr/lib/libblas.so"
In standalone MPASLI, we typically have created environment variables to store this information, so that within a user's environment they simply need to compile MPAS with, e.g., "make gfortran ALBANY=true" and the MPAS_EXTERNAL_LIBS environment variable is automatically included in the build.  However, in ACME it appears that a user's environment is ignored when building ACME, so this approach may not be possible.  One alternative is to specify this information in the machine files for every machine that Albany is supported.  However, I fear this could be hard to maintain.  Also, for non-specific machine files, like linux-generic and mac, it's not clear to me how to implement this in a user-friendly way that does not require the user to manually edit the machine file (or do something complicated like set up modules on a machine that otherwise does not use them).
Andy Salinger, do you have thoughts about how to deal with this?  
Jon Wolfe suggested we look into how Trilinos is currently supported in CESM/ACME.
Note that this gets a little easier to manage if modules for Trilinos and Albany are created, but that does not eliminate the fundamental issue about how to get this environment variable into ACME.  I have created Trilinos and Albany modules for gcc/openmpi on Mustang and Wolf.  These modules set the environment variables required.

Planned Verification and Unit Testing 

Verification and Unit Testing: short-desciption-of-testing-here

Date last modified: 2015/9/4

Contributors: Matt Hoffman

I have implemented and tested items 1 and 2, and they appear to be working. Note I had to remove the dash from the MPAS-LI/mpas-li component name to get the *_CONFIG_OPTS variable to work correctly.  Details of that issue are described here: /wiki/spaces/OCNICE/pages/18939976
But the build dies because MPAS_EXTERNAL_LIBS is not set, so item 3 remains unresolved
libz.so /usr/lib/liblapack.so /usr/lib/libblas.so"

In standalone MPASLI, we typically have created environment variables to store this information, so that within a user's environment they simply need to compile MPAS with, e.g., "make gfortran ALBANY=true" and the MPAS_EXTERNAL_LIBS environment variable is automatically included in the build.  However, in ACME it appears that a user's environment is ignored when building ACME, so this approach is not possible. 

Andy Salinger has recently added files to the Albany build that automatically construct this list of libs and paths.  Because this list is in a standard place (the root of the Albany install), ACME can read this list and add it to the link line in the ACME root Makefile.


4. Setup Albany information for each machine
The above changes still require that a path to Albany is specified for each machine. Each machine needs the path to Albany specified in cime/machines-acme/config_compilers.xml.  If modules are to be loaded, that should happen in cime/machines-acme/env_mach_specific.MACH.  Note that the way things are implemented above, even if the module loads ALBANY_LINK_LIBS or MPAS_EXTERNAL_LIBS, ACME is not using those environment variables - it is creating its own version of ALBANY_LINK_LIBS this way in the Makefile:   include $(ALBANY_PATH)/export_albany.in

Therefore, an Albany module is not required.

Planned Verification and Unit Testing 

Verification and Unit Testing: Verify Working Builds on LCF (and other testing) Machines

Date last modified: 2015/9/4

Contributors: Matt Hoffman, Andy Salinger


Verification testing will consist of:

  1. confirming that ACME can successfully build with MPAS-LI + Albany/Trilinos
  2. confirming that MPAS-LI + Albany/Trilinos can run successfully within ACME
  3. confirming that results from (2) above are consistent with stand-alone runs using MPAS-LI + Albany/Trilinos

Status on Edison:

As of 9/22 we have a successful build of acme/mpasli/albany using the Intel compilers. The pre-existing ACME infrastructure for linking to Trilinos was replicated for Albany. The Albany code was modified to export an environment variable with the complete link line needed to access AlbanyTrilinos, which is then read into the ACME build system. We installed AlbanyTrilinos, compiled with Intel, in the acme project space. There were issues with regard to incompatible netcdf/hdf5 versions between ACME and Albany, and the wrong selection of the C++ compiler in a Makefile, that were the last steps to be sorted out. There is still some clean-up and documentation needed to finish this off.

Status on Titan:

9/25 update: I was successful in linking the acme code to AlbanyTrilinos on Titan. I used GNU for AlbanyTrilinos and PGI for ACME. I had to hand-edit the ALBANY_LINK_LIBS libraries that Albany exports to remove several references to libraries that were not found, and add a specific libstdc++.a that was not the one found by the linker. For both of those modifications, it seems to be an issue of GNU version. PGI seems to provide paths to gcc4.3 libraries while the default gcc compiler is gcc4.8. 

All of this is in my home directory, including inputdata.

Status on Mira:

9/23 update: The XLF compiler suite on Mira does not compile C++11 code in AlbanyTrilinos. We have communicated with ALCF user support, and they claim it is possible to link XLF-compiled Fortran to bgclang++-compiled C++ code. We will attempt this route.

10/15 update: I was successful in linking the acme executable with XLF for fortran and bgclang++ for C++. In the end, I used the C++ compiler to link and had to add several fortran libraries to the link line. The name mangling was different then all other compiler compilations, so had a temporary hack of adding several underscores in a source file. Working on a portable solution. 


Status on LANL HPC:

Albany and Trilinos modules have been created on Mustang and Wolf.  The "MPASLIALB_ONLY" test case builds and runs to completion.  History file output is BFB with the run same configuration in standalone MPAS.  This required the addition of commit 6beb280 (cherry-picked and modified from matthewhoffman/mpasli/lnd_glc_coupling branch) to get the diagnostic solve of velocity occuring on initialization so that evolution on the first time step was correct.


Planned Validation Testing 


Validation testing is not applicable for this task. Verification testing - i.e. that the changes to the build system needed for MPAS-LI in ACME to call the third-party libraries - should be adequate.

Planned Performance Testing


Performance Testing:

Date last modified: 2015/9/22
Contributors: Stephen Price

 

This is part of a new is part of a new land ice model component so there are no previous benchmarks for model performance testing. Performance testing might consist of running ACME in two similar model configurations, one with an inactive (data) land ice model component and one with an active land ice model component so there are no previous benchmarks for model performance testing. See the link to the overal MPAS-Land Ice model integration design document for additional discussion on performance(for which MPAS-LI + Albany/Trilinos are actively calculating ice sheet evolution as part of the coupled climate simulation).