The Design Document page provides a description of the algorithms, implementation and planned testing including unit, verification, validation and performance testing. Please read Step 1.3 Performance Expectations that explains feature documentation requirements from the performance group point of view.
Design Document
| Expand | ||
|---|---|---|
| ||
The first table in Design Document gives overview of this document, from this info the Design Documents Overview page is automatically created. In the table below, 4.Equ means Equations and Algorithms, 5.Ver means Verification, 6.Perf - Performance, 7. Val - Validation
Use the symbols below (copy and paste) to indicate if the section is in progress or done or not started. |
In the table below, 4.Equ means Equations and Algorithms, 5.Ver means Verification, 6.Perf - Performance, 7. Val - Validation, 
- completed, 
- in progress, 
- not done
| Page Properties | ||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||||
|
Title: O_21_LI Third Party Support for MPAS-LI within ACME
Requirements and Design
ACME Ocean and Ice Group
Date:
Summary
The purpose of this section is to summarize what capability is to be added to the ACME Ocean and Ice system through this design process. It should be clear what new code will do that the current code does not. Summarizing the primary challenges with respect to software design and implementation is also appropriate for this section. Finally, this statement should contain general statement with regard to what is “success.”Requirements
Requirement: name-of-requirement-here
Date last modified:
Contributors: Renata McCoy (add your name to this list if it does not appear)
Each requirement is to be listed under a ”section” heading, as there will be a one-to-one correspondence between requirements, design, proposed imple- mentation and testing. Requirements should not discuss technical software issues, but rather focus on model capability. To the extent possible, require- ments should be relatively independent of each other, thus allowing a clean design solution, implementation and testing plan. Algorithmic Formulations
Design solution: short-description-of-proposed-solution-here
Date last modified:
Contributors: (add your name to this list if it does not appear)
For each requirement, there is a design solution that is intended to meet that requirement. Design solutions can include detailed technical discussions of PDEs, algorithms, solvers and similar, as well as technical discussion of performance issues. In general, this section should steer away from a detailed discussion of low-level software issues such as variable declarations, interfaces and sequencing.
Design and Implementation
Implementation: short-desciption-of-implementation-here
Date last modified:
Contributors: (add your name to this list if it does not appear)
This section should detail the plan for implementing the design solution for requirement XXX. In general, this section is software-centric with a focus on software implementation. Pseudo code is appropriate in this section. Links to actual source code are appropriate. Project management items, such as svn branches, timelines and staffing are also appropriate. How do we typeset pseudo code? 9-4-2015
Summary
ACME v1 requires that MPASLI build and run with the First-order solver written in the Albany framework. Albany is a standalone library written in C++ and based on Trilinos. In standalone MPAS, an Albany build is enabled by passing "ALBANY=true" on the build line and defining the needed libraries and their locations with the MPAS_EXTERNAL_LIBS shell variable.
Requirements
Requirement: ACME builds and runs with Albany linked in
Date last modified: 2015/9/4
Contributors: Matt Hoffman, Andy Salinger, Mauro Perego, others?
The model needs to build and run and not be overly cumbersome to use and maintain. MPASLI configurations need to exist with and without Albany.
Algorithmic Formulations
Design and Implementation
Implementation: ACME builds and runs with Albany linked in
Date last modified: 2015/9/4
Contributors: Matt Hoffman, Andy Salinger, Mauro Perego, others?
The basic approach is to duplicate how Trilinos was added to CESM for use with CISM. Trilinos was added to CESM/ACME in a general way that potentially allows any component to use it. Most of the logic is in the highest level Makefile which allows the ACME executable to have the necessary libraries linked in. You can get a sense of what is set up for Trilinos by doing 'git grep -i TRILINOS' from the scripts directory:https://gist.github.com/matthewhoffman/fc46649be672649197fc
Specifically, there are a number of pieces required:
- Use multiple compsets/component names to enable MPASLI with and without Albany.
To enable builds of MPASLI that do and do not include Albany, compsets for each configuration will be created. There will be different component names for MPASLI with and without Albany (e.g., MPASLISIA, MPASLIALBFO), similar to the different versions of CISM that are supported (CISM1, CISM2S, CISM2P, as defined in config_compsets.xml).
2. Use logic and variables in config_compset.xml to indicate needed adjustments to build and namelist generation
echo $MPAS_EXTERNAL_LIBS
-L/home/mhoffman/software/albany/albany-install-mpas/lib -lmpasInterface -lalbanyLib -lalbanySTK -lFELIX -lalbanyAdapt -lalbanySTKRebalance -L/home/mhoffman/software/trilinos/trilinos-git-install/lib -lpiro -lstokhos_muelu -lstokhos_ifpack2 -lstokhos_amesos2 -lstokhos_tpetra -lstokhos_sacado -lstokhos -lrythmos -lmuelu-adapters -lmuelu-interface -lmuelu -llocathyra -llocaepetra -llocalapack -lloca -lnoxepetra -lnoxlapack -lnox -lphalanx -lstk_unit_test_utils -lstk_search -lstk_io_util -lstk_io -lstk_mesh_base -lstk_topology -lstk_util_use_cases -lstk_util_registry -lstk_util_parallel -lstk_util_diag -lstk_util_env -lstk_util_util -lstk_unit_test_utils -lstk_search -lstk_io_util -lstk_io -lstk_mesh_base -lstk_topology -lstk_util_use_cases -lstk_util_registry -lstk_util_parallel -lstk_util_diag -lstk_util_env -lstk_util_util -lintrepid -lteko -lstratimikos -lstratimikosbelos -lstratimikosaztecoo -lstratimikosamesos -lstratimikosml -lstratimikosifpack -lifpack2-adapters -lifpack2 -lzoltan2 -lanasazitpetra -lModeLaplace -lanasaziepetra -lanasazi -lbelostpetra -lbelosepetra -lbelos -lml -lifpack -lmapvarlib -lfastqlib -lblotlib -lplt -lsvdi_cgi -lsvdi_cdr -lsuplib -lsupes -laprepro_lib -lchaco -lIonit -lIotr -lIohb -lIogn -lIopg -lIoexo_fac -lIopx -lIofx -lIoex -lIoss -lnemesis -lexodus_for -lexodus -lmapvarlib -lfastqlib -lblotlib -lplt -lsvdi_cgi -lsvdi_cdr -lsuplib -lsupes -laprepro_lib -lchaco -lIonit -lIotr -lIohb -lIogn -lIopg -lIoexo_fac -lIopx -lIofx -lIoex -lIoss -lnemesis -lexodus_for -lexodus -lpamgen_extras -lpamgen -lamesos2 -lamesos -lgaleri-xpetra -lgaleri -laztecoo -lisorropia -loptipack -lthyratpetra -lthyraepetraext -lthyraepetra -lthyracore -lthyratpetra -lthyraepetraext -lthyraepetra -lthyracore -lxpetra-sup -lxpetra-ext -lxpetra -lepetraext -ltpetraext -ltpetrainout -ltpetra -lkokkostsqr -ltpetrakernels -ltpetraclassiclinalg -ltpetraclassicnodeapi -ltpetraclassic -ltpetraext -ltpetrainout -ltpetra -lkokkostsqr -ltpetrakernels -ltpetraclassiclinalg -ltpetraclassicnodeapi -ltpetraclassic -ltriutils -lglobipack -lshards -lzoltan -lepetra -lsacado -lrtop -lteuchoskokkoscomm -lteuchoskokkoscompat -lteuchosremainder -lteuchosnumerics -lteuchoscomm -lteuchosparameterlist -lteuchoscore -lteuchoskokkoscomm -lteuchoskokkoscompat -lteuchosremainder -lteuchosnumerics -lteuchoscomm -lteuchosparameterlist -lteuchoscore -lkokkosalgorithms -lkokkoscontainers -lkokkoscore -lkokkosalgorithms -lkokkoscontainers -lkokkoscore -ltpi -lgtest /usr/lib/x86_64-linux-gnu/libX11.so /usr/lib/x86_64-linux-gnu/libboost_program_options.so /usr/lib/x86_64-linux-gnu/libboost_system.so /home/mhoffman/software/netcdf/netcdf-4.3.2-install/lib/libnetcdf.so /usr/lib/x86_64-linux-gnu/libhdf5.so /usr/lib/x86_64-linux-gnu/libz.so /usr/lib/liblapack.so /usr/lib/libblas.so -L/usr/lib -lmpi_cxx -lmpi_f90 -lstdc++
export MPAS_EXTERNAL_LIBS="$ALBANY_LIBS $Trilinos_LIBRARY_DIRS $Trilinos_LIBRARIES $Trilinos_TPL_LIBRARIES -L/usr/lib -lmpi_cxx -lmpi_f90 -lstdc++"
export ALBANY_LIBS="-L/home/mhoffman/software/albany/albany-install-mpas/lib -lmpasInterface -lalbanyLib -lalbanySTK -lFELIX -lalbanyAdapt -lalbanySTKRebalance"
## The project library directories.
export Trilinos_LIBRARY_DIRS="-L/home/mhoffman/software/trilinos/trilinos-git-install/lib"
## The project libraries.
export Trilinos_LIBRARIES="-lpiro -lstokhos_muelu -lstokhos_ifpack2 -lstokhos_amesos2 -lstokhos_tpetra -lstokhos_sacado -lstokhos -lrythmos -lmuelu-adapters -lmuelu-interface -lmuelu -llocathyra -llocaepetra -llocalapack -lloca -lnoxepetra -lnoxlapack -lnox -lphalanx -lstk_unit_test_utils -lstk_search -lstk_io_util -lstk_io -lstk_mesh_base -lstk_topology -lstk_util_use_cases -lstk_util_registry -lstk_util_parallel -lstk_util_diag -lstk_util_env -lstk_util_util -lstk_unit_test_utils -lstk_search -lstk_io_util -lstk_io -lstk_mesh_base -lstk_topology -lstk_util_use_cases -lstk_util_registry -lstk_util_parallel -lstk_util_diag -lstk_util_env -lstk_util_util -lintrepid -lteko -lstratimikos -lstratimikosbelos -lstratimikosaztecoo -lstratimikosamesos -lstratimikosml -lstratimikosifpack -lifpack2-adapters -lifpack2 -lzoltan2 -lanasazitpetra -lModeLaplace -lanasaziepetra -lanasazi -lbelostpetra -lbelosepetra -lbelos -lml -lifpack -lmapvarlib -lfastqlib -lblotlib -lplt -lsvdi_cgi -lsvdi_cdr -lsuplib -lsupes -laprepro_lib -lchaco -lIonit -lIotr -lIohb -lIogn -lIopg -lIoexo_fac -lIopx -lIofx -lIoex -lIoss -lnemesis -lexodus_for -lexodus -lmapvarlib -lfastqlib -lblotlib -lplt -lsvdi_cgi -lsvdi_cdr -lsuplib -lsupes -laprepro_lib -lchaco -lIonit -lIotr -lIohb -lIogn -lIopg -lIoexo_fac -lIopx -lIofx -lIoex -lIoss -lnemesis -lexodus_for -lexodus -lpamgen_extras -lpamgen -lamesos2 -lamesos -lgaleri-xpetra -lgaleri -laztecoo -lisorropia -loptipack -lthyratpetra -lthyraepetraext -lthyraepetra -lthyracore -lthyratpetra -lthyraepetraext -lthyraepetra -lthyracore -lxpetra-sup -lxpetra-ext -lxpetra -lepetraext -ltpetraext -ltpetrainout -ltpetra -lkokkostsqr -ltpetrakernels -ltpetraclassiclinalg -ltpetraclassicnodeapi -ltpetraclassic -ltpetraext -ltpetrainout -ltpetra -lkokkostsqr -ltpetrakernels -ltpetraclassiclinalg -ltpetraclassicnodeapi -ltpetraclassic -ltriutils -lglobipack -lshards -lzoltan -lepetra -lsacado -lrtop -lteuchoskokkoscomm -lteuchoskokkoscompat -lteuchosremainder -lteuchosnumerics -lteuchoscomm -lteuchosparameterlist -lteuchoscore -lteuchoskokkoscomm -lteuchoskokkoscompat -lteuchosremainder -lteuchosnumerics -lteuchoscomm -lteuchosparameterlist -lteuchoscore -lkokkosalgorithms -lkokkoscontainers -lkokkoscore -lkokkosalgorithms -lkokkoscontainers -lkokkoscore -ltpi -lgtest"
## The project tpl libraries
export Trilinos_TPL_LIBRARIES="/usr/lib/x86_64-linux-gnu/libX11.so /usr/lib/x86_64-linux-gnu/libboost_program_options.so /usr/lib/x86_64-linux-gnu/libboost_system.so /home/mhoffman/software/netcdf/netcdf-4.3.2-install/lib/libnetcdf.so /usr/lib/x86_64-linux-gnu/libhdf5.so /usr/lib/x86_64-linux-gnu/libz.so /usr/lib/liblapack.so /usr/lib/libblas.so"
Planned Verification and Unit Testing
Verification and Unit Testing:
short-desciption-of-testing-hereVerify Working Builds on LCF (and other testing) Machines
Date last modified:
Contributors: (add your name to this list if it does not appear)
How will XXX be tested? i.e. how will be we know when we have met requirement XXX. Will these unit tests be included in the ongoing going forward?
Planned Validation Testing
Validation Testing: short-desciption-of-testing-here
Date last modified:
Contributors: (add your name to this list if it does not appear)
How will XXX be tested? What observational or other dataset will be used? i.e. how will be we know when we have met requirement XXX. Will these unit tests be included in the ongoing going forward?
Planned Performance Testing
Performance Testing: short-desciption-of-testing-here
Date last modified:
Contributors: (add your name to this list if it does not appear)
How will XXX be tested? i.e. how will be we know when we have met requirement XXX. Will these unit tests be included in the ongoing going forward?
2015/9/4
Contributors: Matt Hoffman, Andy Salinger
Verification testing will consist of:
- confirming that ACME can successfully build with MPAS-LI + Albany/Trilinos
- confirming that MPAS-LI + Albany/Trilinos can run successfully within ACME
- confirming that results from (2) above are consistent with stand-alone runs using MPAS-LI + Albany/Trilinos
Status on Edison:
As of 9/22 we have a successful build of acme/mpasli/albany using the Intel compilers. The pre-existing ACME infrastructure for linking to Trilinos was replicated for Albany. The Albany code was modified to export an environment variable with the complete link line needed to access AlbanyTrilinos, which is then read into the ACME build system. We installed AlbanyTrilinos, compiled with Intel, in the acme project space. There were issues with regard to incompatible netcdf/hdf5 versions between ACME and Albany, and the wrong selection of the C++ compiler in a Makefile, that were the last steps to be sorted out. There is still some clean-up and documentation needed to finish this off.
Status on Titan:
9/25 update: I was successful in linking the acme code to AlbanyTrilinos on Titan. I used GNU for AlbanyTrilinos and PGI for ACME. I had to hand-edit the ALBANY_LINK_LIBS libraries that Albany exports to remove several references to libraries that were not found, and add a specific libstdc++.a that was not the one found by the linker. For both of those modifications, it seems to be an issue of GNU version. PGI seems to provide paths to gcc4.3 libraries while the default gcc compiler is gcc4.8.
All of this is in my home directory, including inputdata.
Status on Mira:
9/23 update: The XLF compiler suite on Mira does not compile C++11 code in AlbanyTrilinos. We have communicated with ALCF user support, and they claim it is possible to link XLF-compiled Fortran to bgclang++-compiled C++ code. We will attempt this route.
10/15 update: I was successful in linking the acme executable with XLF for fortran and bgclang++ for C++. In the end, I used the C++ compiler to link and had to add several fortran libraries to the link line. The name mangling was different then all other compiler compilations, so had a temporary hack of adding several underscores in a source file. Working on a portable solution.
Status on LANL HPC:
Albany and Trilinos modules have been created on Mustang and Wolf. The "MPASLIALB_ONLY" test case builds and runs to completion. History file output is BFB with the run same configuration in standalone MPAS. This required the addition of commit 6beb280 (cherry-picked and modified from matthewhoffman/mpasli/lnd_glc_coupling branch) to get the diagnostic solve of velocity occuring on initialization so that evolution on the first time step was correct.
Planned Validation Testing
Planned Performance Testing
Date last modified: 2015/9/22
Contributors: Stephen Price
This is part of a new land ice model component so there are no previous benchmarks for model performance testing. Performance testing might consist of running ACME in two similar model configurations, one with an inactive (data) land ice model component and one with an active land ice model component (for which MPAS-LI + Albany/Trilinos are actively calculating ice sheet evolution as part of the coupled climate simulation).