The Design Document page provides a description of the algorithms, implementation and planned testing including unit, verification, validation and performance testing. Please read Step 1.3 Performance Expectations that explains feature documentation requirements from the performance group point of view.
Design Document
Expand | ||
---|---|---|
| ||
The first table in Design Document gives overview of this document, from this info the Design Documents Overview page is automatically created. In the table below, 4.Equ means Equations and Algorithms, 5.Ver means Verification, 6.Perf - Performance, 7. Val - Validation
Use the symbols below (copy and paste) to indicate if the section is in progress or done or not started. |
In the table below, 4.Equ means Equations and Algorithms, 5.Ver means Verification, 6.Perf - Performance, 7. Val - Validation, - completed, - in progress, - not done
Page Properties | ||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||
|
Title: O_21_LI Third Party Support for MPAS-LI within ACME
Requirements and Design
ACME Ocean and Ice Group
Date: 9-4-2015
Summary
ACME v1 requires that MPASLI build and run with the First-order solver written in the Albany framework. Albany is a standalone library written in C++ and based on Trilinos. In standalone MPAS, an Albany build is enabled by passing "ALBANY=true" on the build line and defining the needed libraries and their locations with the MPAS_EXTERNAL_LIBS shell variable.
Requirements
Requirement: ACME builds and runs with Albany linked in
Date last modified: 2015/9/4
Contributors: Matt Hoffman, Andy Salinger, Mauro Perego, others?
The model needs to build and run and not be overly cumbersome to use and maintain. MPASLI configurations need to exist with and without Albany.
Algorithmic Formulations
Design and Implementation
Implementation: ACME builds and runs with Albany linked in
Date last modified: 2015/9/4
Contributors: Matt Hoffman, Andy Salinger, Mauro Perego, others?
The basic approach is to duplicate how Trilinos was added to CESM for use with CISM. Trilinos was added to CESM/ACME in a general way that potentially allows any component to use it. Most of the logic is in the highest level Makefile which allows the ACME executable to have the necessary libraries linked in. You can get a sense of what is set up for Trilinos by doing 'git grep -i TRILINOS' from the scripts directory:https://gist.github.com/matthewhoffman/fc46649be672649197fc
Specifically, there are a number of pieces required:
- Use multiple compsets/component names to enable MPASLI with and without Albany.
To enable builds of MPASLI that do and do not include Albany, compsets for each configuration will be created. There will be different component names for MPASLI with and without Albany (e.g., MPASLISIA, MPASLIALBFO), similar to the different versions of CISM that are supported (CISM1, CISM2S, CISM2P, as defined in config_compsets.xml).
2. Use logic and variables in config_compset.xml to indicate needed adjustments to build and namelist generation
echo $MPAS_EXTERNAL_LIBS
-L/home/mhoffman/software/albany/albany-install-mpas/lib -lmpasInterface -lalbanyLib -lalbanySTK -lFELIX -lalbanyAdapt -lalbanySTKRebalance -L/home/mhoffman/software/trilinos/trilinos-git-install/lib -lpiro -lstokhos_muelu -lstokhos_ifpack2 -lstokhos_amesos2 -lstokhos_tpetra -lstokhos_sacado -lstokhos -lrythmos -lmuelu-adapters -lmuelu-interface -lmuelu -llocathyra -llocaepetra -llocalapack -lloca -lnoxepetra -lnoxlapack -lnox -lphalanx -lstk_unit_test_utils -lstk_search -lstk_io_util -lstk_io -lstk_mesh_base -lstk_topology -lstk_util_use_cases -lstk_util_registry -lstk_util_parallel -lstk_util_diag -lstk_util_env -lstk_util_util -lstk_unit_test_utils -lstk_search -lstk_io_util -lstk_io -lstk_mesh_base -lstk_topology -lstk_util_use_cases -lstk_util_registry -lstk_util_parallel -lstk_util_diag -lstk_util_env -lstk_util_util -lintrepid -lteko -lstratimikos -lstratimikosbelos -lstratimikosaztecoo -lstratimikosamesos -lstratimikosml -lstratimikosifpack -lifpack2-adapters -lifpack2 -lzoltan2 -lanasazitpetra -lModeLaplace -lanasaziepetra -lanasazi -lbelostpetra -lbelosepetra -lbelos -lml -lifpack -lmapvarlib -lfastqlib -lblotlib -lplt -lsvdi_cgi -lsvdi_cdr -lsuplib -lsupes -laprepro_lib -lchaco -lIonit -lIotr -lIohb -lIogn -lIopg -lIoexo_fac -lIopx -lIofx -lIoex -lIoss -lnemesis -lexodus_for -lexodus -lmapvarlib -lfastqlib -lblotlib -lplt -lsvdi_cgi -lsvdi_cdr -lsuplib -lsupes -laprepro_lib -lchaco -lIonit -lIotr -lIohb -lIogn -lIopg -lIoexo_fac -lIopx -lIofx -lIoex -lIoss -lnemesis -lexodus_for -lexodus -lpamgen_extras -lpamgen -lamesos2 -lamesos -lgaleri-xpetra -lgaleri -laztecoo -lisorropia -loptipack -lthyratpetra -lthyraepetraext -lthyraepetra -lthyracore -lthyratpetra -lthyraepetraext -lthyraepetra -lthyracore -lxpetra-sup -lxpetra-ext -lxpetra -lepetraext -ltpetraext -ltpetrainout -ltpetra -lkokkostsqr -ltpetrakernels -ltpetraclassiclinalg -ltpetraclassicnodeapi -ltpetraclassic -ltpetraext -ltpetrainout -ltpetra -lkokkostsqr -ltpetrakernels -ltpetraclassiclinalg -ltpetraclassicnodeapi -ltpetraclassic -ltriutils -lglobipack -lshards -lzoltan -lepetra -lsacado -lrtop -lteuchoskokkoscomm -lteuchoskokkoscompat -lteuchosremainder -lteuchosnumerics -lteuchoscomm -lteuchosparameterlist -lteuchoscore -lteuchoskokkoscomm -lteuchoskokkoscompat -lteuchosremainder -lteuchosnumerics -lteuchoscomm -lteuchosparameterlist -lteuchoscore -lkokkosalgorithms -lkokkoscontainers -lkokkoscore -lkokkosalgorithms -lkokkoscontainers -lkokkoscore -ltpi -lgtest /usr/lib/x86_64-linux-gnu/libX11.so /usr/lib/x86_64-linux-gnu/libboost_program_options.so /usr/lib/x86_64-linux-gnu/libboost_system.so /home/mhoffman/software/netcdf/netcdf-4.3.2-install/lib/libnetcdf.so /usr/lib/x86_64-linux-gnu/libhdf5.so /usr/lib/x86_64-linux-gnu/libz.so /usr/lib/liblapack.so /usr/lib/libblas.so -L/usr/lib -lmpi_cxx -lmpi_f90 -lstdc++
export MPAS_EXTERNAL_LIBS="$ALBANY_LIBS $Trilinos_LIBRARY_DIRS $Trilinos_LIBRARIES $Trilinos_TPL_LIBRARIES -L/usr/lib -lmpi_cxx -lmpi_f90 -lstdc++"
export ALBANY_LIBS="-L/home/mhoffman/software/albany/albany-install-mpas/lib -lmpasInterface -lalbanyLib -lalbanySTK -lFELIX -lalbanyAdapt -lalbanySTKRebalance"
## The project library directories.
export Trilinos_LIBRARY_DIRS="-L/home/mhoffman/software/trilinos/trilinos-git-install/lib"
## The project libraries.
export Trilinos_LIBRARIES="-lpiro -lstokhos_muelu -lstokhos_ifpack2 -lstokhos_amesos2 -lstokhos_tpetra -lstokhos_sacado -lstokhos -lrythmos -lmuelu-adapters -lmuelu-interface -lmuelu -llocathyra -llocaepetra -llocalapack -lloca -lnoxepetra -lnoxlapack -lnox -lphalanx -lstk_unit_test_utils -lstk_search -lstk_io_util -lstk_io -lstk_mesh_base -lstk_topology -lstk_util_use_cases -lstk_util_registry -lstk_util_parallel -lstk_util_diag -lstk_util_env -lstk_util_util -lstk_unit_test_utils -lstk_search -lstk_io_util -lstk_io -lstk_mesh_base -lstk_topology -lstk_util_use_cases -lstk_util_registry -lstk_util_parallel -lstk_util_diag -lstk_util_env -lstk_util_util -lintrepid -lteko -lstratimikos -lstratimikosbelos -lstratimikosaztecoo -lstratimikosamesos -lstratimikosml -lstratimikosifpack -lifpack2-adapters -lifpack2 -lzoltan2 -lanasazitpetra -lModeLaplace -lanasaziepetra -lanasazi -lbelostpetra -lbelosepetra -lbelos -lml -lifpack -lmapvarlib -lfastqlib -lblotlib -lplt -lsvdi_cgi -lsvdi_cdr -lsuplib -lsupes -laprepro_lib -lchaco -lIonit -lIotr -lIohb -lIogn -lIopg -lIoexo_fac -lIopx -lIofx -lIoex -lIoss -lnemesis -lexodus_for -lexodus -lmapvarlib -lfastqlib -lblotlib -lplt -lsvdi_cgi -lsvdi_cdr -lsuplib -lsupes -laprepro_lib -lchaco -lIonit -lIotr -lIohb -lIogn -lIopg -lIoexo_fac -lIopx -lIofx -lIoex -lIoss -lnemesis -lexodus_for -lexodus -lpamgen_extras -lpamgen -lamesos2 -lamesos -lgaleri-xpetra -lgaleri -laztecoo -lisorropia -loptipack -lthyratpetra -lthyraepetraext -lthyraepetra -lthyracore -lthyratpetra -lthyraepetraext -lthyraepetra -lthyracore -lxpetra-sup -lxpetra-ext -lxpetra -lepetraext -ltpetraext -ltpetrainout -ltpetra -lkokkostsqr -ltpetrakernels -ltpetraclassiclinalg -ltpetraclassicnodeapi -ltpetraclassic -ltpetraext -ltpetrainout -ltpetra -lkokkostsqr -ltpetrakernels -ltpetraclassiclinalg -ltpetraclassicnodeapi -ltpetraclassic -ltriutils -lglobipack -lshards -lzoltan -lepetra -lsacado -lrtop -lteuchoskokkoscomm -lteuchoskokkoscompat -lteuchosremainder -lteuchosnumerics -lteuchoscomm -lteuchosparameterlist -lteuchoscore -lteuchoskokkoscomm -lteuchoskokkoscompat -lteuchosremainder -lteuchosnumerics -lteuchoscomm -lteuchosparameterlist -lteuchoscore -lkokkosalgorithms -lkokkoscontainers -lkokkoscore -lkokkosalgorithms -lkokkoscontainers -lkokkoscore -ltpi -lgtest"
## The project tpl libraries
export Trilinos_TPL_LIBRARIES="/usr/lib/x86_64-linux-gnu/libX11.so /usr/lib/x86_64-linux-gnu/libboost_program_options.so /usr/lib/x86_64-linux-gnu/libboost_system.so /home/mhoffman/software/netcdf/netcdf-4.3.2-install/lib/libnetcdf.so /usr/lib/x86_64-linux-gnu/libhdf5.so /usr/lib/x86_64-linux-gnu/libz.so /usr/lib/liblapack.so /usr/lib/libblas.so"
Planned Verification and Unit Testing
Verification and Unit Testing: Verify Working Builds on LCF (and other testing) Machines
Date last modified: 2015/9/4
Contributors: Matt Hoffman, Andy Salinger
Status on Edison:
As of 9/22 we have a successful build of acme/mpasli/albany using the Intel compilers. The pre-existing ACME infrastructure for linking to Trilinos was replicated for Albany. The Albany code was modified to export an environment variable with the complete link line needed to access AlbanyTrilinos, which is then read into the ACME build system. We installed AlbanyTrilinos, compiled with Intel, in the acme project space. There were issues with regard to incompatible netcdf/hdf5 versions between ACME and Albany, and the wrong selection of the C++ compiler in a Makefile, that were the last steps to be sorted out. There is still some clean-up and documentation needed to finish this off.
Status on Titan:
9/23 update: The PGI compilers on Titan do not compile C++ code in AlbanyTrilinos that uses the newish C++11 standard. There is documentation online at OLCF which says that PGI and GNU produce binary compatible libraries. We have successfully compiled AlbanyTrilinos with GNU. The next step is to attempt to compile ACME and/or MPASLI with PGI and link to this installed Albany.
Status on Mira:
9/23 update: The XLF compiler suite on Mira does not compile C++11 code in AlbanyTrilinos. We have communicated with ALCF user support, and they claim it is possible to link XLF-compiled Fortran to bgclang++-compiled C++ code. We will attempt this route.
Albany and Trilinos modules have been created on Mustang and Wolf. The test branch can compile ACME with Albany included.
Status on LANL HPC:
Albany and Trilinos modules have been created on Mustang and Wolf. The test branch can compile ACME with Albany included.
Planned Validation Testing
Planned Performance Testing
Date last modified: 2015/9/22
Contributors: Stephen Price
This is part of a new land ice model component so there are no previous benchmarks for model performance testing. See the link to the overal MPAS-Land Ice model integration design document for additional discussion on performance.