O_21_LI Third Party Support for MPAS-LI within ACME Design Document
The Design Document page provides a description of the algorithms, implementation and planned testing including unit, verification, validation and performance testing. Please read Step 1.3 Performance Expectations that explains feature documentation requirements from the performance group point of view.
Design Document
In the table below, 4.Equ means Equations and Algorithms, 5.Ver means Verification, 6.Perf - Performance, 7. Val - Validation, - completed, - in progress, - not done
Title: O_21_LI Third Party Support for MPAS-LI within ACME
Requirements and Design
ACME Ocean and Ice Group
Date: 9-4-2015
Summary
ACME v1 requires that MPASLI build and run with the First-order solver written in the Albany framework. Albany is a standalone library written in C++ and based on Trilinos. In standalone MPAS, an Albany build is enabled by passing "ALBANY=true" on the build line and defining the needed libraries and their locations with the MPAS_EXTERNAL_LIBS shell variable.
Requirements
Requirement: ACME builds and runs with Albany linked in
Date last modified: 2015/9/4
Contributors: Matt Hoffman, Andy Salinger, Mauro Perego, others?
The model needs to build and run and not be overly cumbersome to use and maintain. MPASLI configurations need to exist with and without Albany.
Algorithmic Formulations
Design and Implementation
Implementation: ACME builds and runs with Albany linked in
Date last modified: 2015/9/4
Contributors: Matt Hoffman, Andy Salinger, Mauro Perego, others?
The basic approach is to duplicate how Trilinos was added to CESM for use with CISM. Trilinos was added to CESM/ACME in a general way that potentially allows any component to use it. Most of the logic is in the highest level Makefile which allows the ACME executable to have the necessary libraries linked in. You can get a sense of what is set up for Trilinos by doing 'git grep -i TRILINOS' from the scripts directory:https://gist.github.com/matthewhoffman/fc46649be672649197fc
Specifically, there are a number of pieces required:
- Use multiple compsets/component names to enable MPASLI with and without Albany.
To enable builds of MPASLI that do and do not include Albany, compsets for each configuration will be created. There will be different component names for MPASLI with and without Albany (e.g., MPASLISIA, MPASLIALBFO), similar to the different versions of CISM that are supported (CISM1, CISM2S, CISM2P, as defined in config_compsets.xml).
2. Use logic and variables in config_compset.xml to indicate needed adjustments to build and namelist generation
echo $MPAS_EXTERNAL_LIBS
-L/home/mhoffman/software/albany/albany-install-mpas/lib -lmpasInterface -lalbanyLib -lalbanySTK -lFELIX -lalbanyAdapt -lalbanySTKRebalance -L/home/mhoffman/software/trilinos/trilinos-git-install/lib -lpiro -lstokhos_muelu -lstokhos_ifpack2 -lstokhos_amesos2 -lstokhos_tpetra -lstokhos_sacado -lstokhos -lrythmos -lmuelu-adapters -lmuelu-interface -lmuelu -llocathyra -llocaepetra -llocalapack -lloca -lnoxepetra -lnoxlapack -lnox -lphalanx -lstk_unit_test_utils -lstk_search -lstk_io_util -lstk_io -lstk_mesh_base -lstk_topology -lstk_util_use_cases -lstk_util_registry -lstk_util_parallel -lstk_util_diag -lstk_util_env -lstk_util_util -lstk_unit_test_utils -lstk_search -lstk_io_util -lstk_io -lstk_mesh_base -lstk_topology -lstk_util_use_cases -lstk_util_registry -lstk_util_parallel -lstk_util_diag -lstk_util_env -lstk_util_util -lintrepid -lteko -lstratimikos -lstratimikosbelos -lstratimikosaztecoo -lstratimikosamesos -lstratimikosml -lstratimikosifpack -lifpack2-adapters -lifpack2 -lzoltan2 -lanasazitpetra -lModeLaplace -lanasaziepetra -lanasazi -lbelostpetra -lbelosepetra -lbelos -lml -lifpack -lmapvarlib -lfastqlib -lblotlib -lplt -lsvdi_cgi -lsvdi_cdr -lsuplib -lsupes -laprepro_lib -lchaco -lIonit -lIotr -lIohb -lIogn -lIopg -lIoexo_fac -lIopx -lIofx -lIoex -lIoss -lnemesis -lexodus_for -lexodus -lmapvarlib -lfastqlib -lblotlib -lplt -lsvdi_cgi -lsvdi_cdr -lsuplib -lsupes -laprepro_lib -lchaco -lIonit -lIotr -lIohb -lIogn -lIopg -lIoexo_fac -lIopx -lIofx -lIoex -lIoss -lnemesis -lexodus_for -lexodus -lpamgen_extras -lpamgen -lamesos2 -lamesos -lgaleri-xpetra -lgaleri -laztecoo -lisorropia -loptipack -lthyratpetra -lthyraepetraext -lthyraepetra -lthyracore -lthyratpetra -lthyraepetraext -lthyraepetra -lthyracore -lxpetra-sup -lxpetra-ext -lxpetra -lepetraext -ltpetraext -ltpetrainout -ltpetra -lkokkostsqr -ltpetrakernels -ltpetraclassiclinalg -ltpetraclassicnodeapi -ltpetraclassic -ltpetraext -ltpetrainout -ltpetra -lkokkostsqr -ltpetrakernels -ltpetraclassiclinalg -ltpetraclassicnodeapi -ltpetraclassic -ltriutils -lglobipack -lshards -lzoltan -lepetra -lsacado -lrtop -lteuchoskokkoscomm -lteuchoskokkoscompat -lteuchosremainder -lteuchosnumerics -lteuchoscomm -lteuchosparameterlist -lteuchoscore -lteuchoskokkoscomm -lteuchoskokkoscompat -lteuchosremainder -lteuchosnumerics -lteuchoscomm -lteuchosparameterlist -lteuchoscore -lkokkosalgorithms -lkokkoscontainers -lkokkoscore -lkokkosalgorithms -lkokkoscontainers -lkokkoscore -ltpi -lgtest /usr/lib/x86_64-linux-gnu/libX11.so /usr/lib/x86_64-linux-gnu/libboost_program_options.so /usr/lib/x86_64-linux-gnu/libboost_system.so /home/mhoffman/software/netcdf/netcdf-4.3.2-install/lib/libnetcdf.so /usr/lib/x86_64-linux-gnu/libhdf5.so /usr/lib/x86_64-linux-gnu/libz.so /usr/lib/liblapack.so /usr/lib/libblas.so -L/usr/lib -lmpi_cxx -lmpi_f90 -lstdc++
export MPAS_EXTERNAL_LIBS="$ALBANY_LIBS $Trilinos_LIBRARY_DIRS $Trilinos_LIBRARIES $Trilinos_TPL_LIBRARIES -L/usr/lib -lmpi_cxx -lmpi_f90 -lstdc++"
export ALBANY_LIBS="-L/home/mhoffman/software/albany/albany-install-mpas/lib -lmpasInterface -lalbanyLib -lalbanySTK -lFELIX -lalbanyAdapt -lalbanySTKRebalance"
## The project library directories.
export Trilinos_LIBRARY_DIRS="-L/home/mhoffman/software/trilinos/trilinos-git-install/lib"
## The project libraries.
export Trilinos_LIBRARIES="-lpiro -lstokhos_muelu -lstokhos_ifpack2 -lstokhos_amesos2 -lstokhos_tpetra -lstokhos_sacado -lstokhos -lrythmos -lmuelu-adapters -lmuelu-interface -lmuelu -llocathyra -llocaepetra -llocalapack -lloca -lnoxepetra -lnoxlapack -lnox -lphalanx -lstk_unit_test_utils -lstk_search -lstk_io_util -lstk_io -lstk_mesh_base -lstk_topology -lstk_util_use_cases -lstk_util_registry -lstk_util_parallel -lstk_util_diag -lstk_util_env -lstk_util_util -lstk_unit_test_utils -lstk_search -lstk_io_util -lstk_io -lstk_mesh_base -lstk_topology -lstk_util_use_cases -lstk_util_registry -lstk_util_parallel -lstk_util_diag -lstk_util_env -lstk_util_util -lintrepid -lteko -lstratimikos -lstratimikosbelos -lstratimikosaztecoo -lstratimikosamesos -lstratimikosml -lstratimikosifpack -lifpack2-adapters -lifpack2 -lzoltan2 -lanasazitpetra -lModeLaplace -lanasaziepetra -lanasazi -lbelostpetra -lbelosepetra -lbelos -lml -lifpack -lmapvarlib -lfastqlib -lblotlib -lplt -lsvdi_cgi -lsvdi_cdr -lsuplib -lsupes -laprepro_lib -lchaco -lIonit -lIotr -lIohb -lIogn -lIopg -lIoexo_fac -lIopx -lIofx -lIoex -lIoss -lnemesis -lexodus_for -lexodus -lmapvarlib -lfastqlib -lblotlib -lplt -lsvdi_cgi -lsvdi_cdr -lsuplib -lsupes -laprepro_lib -lchaco -lIonit -lIotr -lIohb -lIogn -lIopg -lIoexo_fac -lIopx -lIofx -lIoex -lIoss -lnemesis -lexodus_for -lexodus -lpamgen_extras -lpamgen -lamesos2 -lamesos -lgaleri-xpetra -lgaleri -laztecoo -lisorropia -loptipack -lthyratpetra -lthyraepetraext -lthyraepetra -lthyracore -lthyratpetra -lthyraepetraext -lthyraepetra -lthyracore -lxpetra-sup -lxpetra-ext -lxpetra -lepetraext -ltpetraext -ltpetrainout -ltpetra -lkokkostsqr -ltpetrakernels -ltpetraclassiclinalg -ltpetraclassicnodeapi -ltpetraclassic -ltpetraext -ltpetrainout -ltpetra -lkokkostsqr -ltpetrakernels -ltpetraclassiclinalg -ltpetraclassicnodeapi -ltpetraclassic -ltriutils -lglobipack -lshards -lzoltan -lepetra -lsacado -lrtop -lteuchoskokkoscomm -lteuchoskokkoscompat -lteuchosremainder -lteuchosnumerics -lteuchoscomm -lteuchosparameterlist -lteuchoscore -lteuchoskokkoscomm -lteuchoskokkoscompat -lteuchosremainder -lteuchosnumerics -lteuchoscomm -lteuchosparameterlist -lteuchoscore -lkokkosalgorithms -lkokkoscontainers -lkokkoscore -lkokkosalgorithms -lkokkoscontainers -lkokkoscore -ltpi -lgtest"
## The project tpl libraries
export Trilinos_TPL_LIBRARIES="/usr/lib/x86_64-linux-gnu/libX11.so /usr/lib/x86_64-linux-gnu/libboost_program_options.so /usr/lib/x86_64-linux-gnu/libboost_system.so /home/mhoffman/software/netcdf/netcdf-4.3.2-install/lib/libnetcdf.so /usr/lib/x86_64-linux-gnu/libhdf5.so /usr/lib/x86_64-linux-gnu/libz.so /usr/lib/liblapack.so /usr/lib/libblas.so"
Planned Verification and Unit Testing
Verification and Unit Testing: Verify Working Builds on LCF (and other testing) Machines
Date last modified: 2015/9/4
Contributors: Matt Hoffman, Andy Salinger
Verification testing will consist of:
- confirming that ACME can successfully build with MPAS-LI + Albany/Trilinos
- confirming that MPAS-LI + Albany/Trilinos can run successfully within ACME
- confirming that results from (2) above are consistent with stand-alone runs using MPAS-LI + Albany/Trilinos
Status on Edison:
As of 9/22 we have a successful build of acme/mpasli/albany using the Intel compilers. The pre-existing ACME infrastructure for linking to Trilinos was replicated for Albany. The Albany code was modified to export an environment variable with the complete link line needed to access AlbanyTrilinos, which is then read into the ACME build system. We installed AlbanyTrilinos, compiled with Intel, in the acme project space. There were issues with regard to incompatible netcdf/hdf5 versions between ACME and Albany, and the wrong selection of the C++ compiler in a Makefile, that were the last steps to be sorted out. There is still some clean-up and documentation needed to finish this off.
Status on Titan:
9/25 update: I was successful in linking the acme code to AlbanyTrilinos on Titan. I used GNU for AlbanyTrilinos and PGI for ACME. I had to hand-edit the ALBANY_LINK_LIBS libraries that Albany exports to remove several references to libraries that were not found, and add a specific libstdc++.a that was not the one found by the linker. For both of those modifications, it seems to be an issue of GNU version. PGI seems to provide paths to gcc4.3 libraries while the default gcc compiler is gcc4.8.
All of this is in my home directory, including inputdata.
Status on Mira:
9/23 update: The XLF compiler suite on Mira does not compile C++11 code in AlbanyTrilinos. We have communicated with ALCF user support, and they claim it is possible to link XLF-compiled Fortran to bgclang++-compiled C++ code. We will attempt this route.
10/15 update: I was successful in linking the acme executable with XLF for fortran and bgclang++ for C++. In the end, I used the C++ compiler to link and had to add several fortran libraries to the link line. The name mangling was different then all other compiler compilations, so had a temporary hack of adding several underscores in a source file. Working on a portable solution.
Status on LANL HPC:
Albany and Trilinos modules have been created on Mustang and Wolf. The "MPASLIALB_ONLY" test case builds and runs to completion. History file output is BFB with the run same configuration in standalone MPAS. This required the addition of commit 6beb280 (cherry-picked and modified from matthewhoffman/mpasli/lnd_glc_coupling branch) to get the diagnostic solve of velocity occuring on initialization so that evolution on the first time step was correct.
Planned Validation Testing
Planned Performance Testing
Date last modified: 2015/9/22
Contributors: Stephen Price
This is part of a new land ice model component so there are no previous benchmarks for model performance testing. Performance testing might consist of running ACME in two similar model configurations, one with an inactive (data) land ice model component and one with an active land ice model component (for which MPAS-LI + Albany/Trilinos are actively calculating ice sheet evolution as part of the coupled climate simulation).