Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 10 Next »

The Design Document page provides a description of the algorithms, implementation and planned testing including unit, verification, validation and performance testing. Please read  Step 1.3 Performance Expectations that explains feature documentation requirements from the performance group point of view. 

Design Document

 

 Click here for instructions to fill up the table below ......

The first table in Design Document gives overview of this document, from this info the Design Documents Overview page is automatically created.

In the table below, 4.Equ means Equations and Algorithms, 5.Ver means Verification, 6.Perf - Performance, 7. Val - Validation

  • Equations: Document the equations that are being solved and describe algorithms
  • Verification Plans: Define tests that will be run to show that implementation is correct and robust. Involve unit tests to cover range of inputs as well as benchmarks.
  • Performance expectations: Explain the expected performance impact from this development
  • Validation Plans: Document what process-based, stand-alone component, and coupled model runs will be performed, and with what metrics will be used to assess validity

Use the symbols below (copy and paste) to indicate if the section is in progress or done or not started.

In the table below, 4.Equ means Equations and Algorithms, 5.Ver means Verification, 6.Perf - Performance, 7. Val - Validation,   (tick) - completed, (warning) - in progress, (error) - not done

 

Overview table for the owner and an approver of this feature

1.Description

O_21_LI Third Party Support for MPAS-LI within ACME
2.OwnerAndy Salinger
3.Created 
4.EquN/A (build system modifications)
5.Ver(warning)
6.Perf(warning)
7.Val(warning)
8.Approver 
9.Approved Date 

 

 Click here for Table of Contents ...

Table of Contents

 

 

 


Title: O_21_LI Third Party Support for MPAS-LI within ACME

Requirements and Design

ACME Ocean and Ice Group

Date:  9-4-2015

Summary

ACME v1 requires that MPASLI build and run with the First-order solver written in the Albany framework.  Albany is a standalone library written in C++ and based on Trilinos.  In standalone MPAS, an Albany build is enabled by passing "ALBANY=true" on the build line and defining the needed libraries and their locations with the MPAS_EXTERNAL_LIBS shell variable.

Requirements


Requirement: ACME builds and runs with Albany linked in

Date last modified: 2015/9/4
Contributors: Matt Hoffman, Andy Salinger, Mauro Perego, others?

The model needs to build and run and not be overly cumbersome to use and maintain.  MPASLI configurations need to exist with and without Albany.

 

Algorithmic Formulations


There are no algorithmic formulations for this task as it pertains only to build system modifications.

Design and Implementation

Implementation: ACME builds and runs with Albany linked in

Date last modified: 2015/9/4

Contributors: Matt Hoffman, Andy Salinger, Mauro Perego, others?


  1. Use multiple compsets/component names to enable MPASLI with and without Albany.

To enable builds of MPASLI that do and do not include Albany, compsets for each configuration will be created.  There will be different component names for MPASLI with and without Albany (e.g., MPASLISIA, MPASLIALBFO), similar to the different versions of CISM that are supported (CISM1, CISM2S, CISM2P, as defined in config_compsets.xml). 

 

2. Use *_CONFIG_OPTS variable to indicate needed adjustments to build and namelist generation

The *_CONFIG_OPTS that is automatically parsed for each component will be used to enable the HO physics with Albany:

<MPASLI_CONFIG_OPTS compset="_MPASLIALBFO">-phys fo</MPASLI_CONFIG_OPTS>

The MPASLI_CONFIG_OPTS variable can then be used at build time in models/glc/mpasli/bld/mpasli.buildexe.csh to add the "ALBANY=true" compile flag for MPAS.
This can also be used to adjust the namelist options to select the Albany velocity solver for these compsets.
3. Set MPAS_EXTERNAL_LIBS shell variable and give the build system access to it

MPAS_EXTERNAL_LIBS is the variable that indicates the libraries and paths for the Albany and Trilinos (and a few other things like MPI, boost, etc.) libraries.  Right now on a generic linux machine it looks something like this:
echo $MPAS_EXTERNAL_LIBS
-L/home/mhoffman/software/albany/albany-install-mpas/lib -lmpasInterface -lalbanyLib -lalbanySTK -lFELIX -lalbanyAdapt -lalbanySTKRebalance -L/home/mhoffman/software/trilinos/trilinos-git-install/lib -lpiro -lstokhos_muelu -lstokhos_ifpack2 -lstokhos_amesos2 -lstokhos_tpetra -lstokhos_sacado -lstokhos -lrythmos -lmuelu-adapters -lmuelu-interface -lmuelu -llocathyra -llocaepetra -llocalapack -lloca -lnoxepetra -lnoxlapack -lnox -lphalanx -lstk_unit_test_utils -lstk_search -lstk_io_util -lstk_io -lstk_mesh_base -lstk_topology -lstk_util_use_cases -lstk_util_registry -lstk_util_parallel -lstk_util_diag -lstk_util_env -lstk_util_util -lstk_unit_test_utils -lstk_search -lstk_io_util -lstk_io -lstk_mesh_base -lstk_topology -lstk_util_use_cases -lstk_util_registry -lstk_util_parallel -lstk_util_diag -lstk_util_env -lstk_util_util -lintrepid -lteko -lstratimikos -lstratimikosbelos -lstratimikosaztecoo -lstratimikosamesos -lstratimikosml -lstratimikosifpack -lifpack2-adapters -lifpack2 -lzoltan2 -lanasazitpetra -lModeLaplace -lanasaziepetra -lanasazi -lbelostpetra -lbelosepetra -lbelos -lml -lifpack -lmapvarlib -lfastqlib -lblotlib -lplt -lsvdi_cgi -lsvdi_cdr -lsuplib -lsupes -laprepro_lib -lchaco -lIonit -lIotr -lIohb -lIogn -lIopg -lIoexo_fac -lIopx -lIofx -lIoex -lIoss -lnemesis -lexodus_for -lexodus -lmapvarlib -lfastqlib -lblotlib -lplt -lsvdi_cgi -lsvdi_cdr -lsuplib -lsupes -laprepro_lib -lchaco -lIonit -lIotr -lIohb -lIogn -lIopg -lIoexo_fac -lIopx -lIofx -lIoex -lIoss -lnemesis -lexodus_for -lexodus -lpamgen_extras -lpamgen -lamesos2 -lamesos -lgaleri-xpetra -lgaleri -laztecoo -lisorropia -loptipack -lthyratpetra -lthyraepetraext -lthyraepetra -lthyracore -lthyratpetra -lthyraepetraext -lthyraepetra -lthyracore -lxpetra-sup -lxpetra-ext -lxpetra -lepetraext -ltpetraext -ltpetrainout -ltpetra -lkokkostsqr -ltpetrakernels -ltpetraclassiclinalg -ltpetraclassicnodeapi -ltpetraclassic -ltpetraext -ltpetrainout -ltpetra -lkokkostsqr -ltpetrakernels -ltpetraclassiclinalg -ltpetraclassicnodeapi -ltpetraclassic -ltriutils -lglobipack -lshards -lzoltan -lepetra -lsacado -lrtop -lteuchoskokkoscomm -lteuchoskokkoscompat -lteuchosremainder -lteuchosnumerics -lteuchoscomm -lteuchosparameterlist -lteuchoscore -lteuchoskokkoscomm -lteuchoskokkoscompat -lteuchosremainder -lteuchosnumerics -lteuchoscomm -lteuchosparameterlist -lteuchoscore -lkokkosalgorithms -lkokkoscontainers -lkokkoscore -lkokkosalgorithms -lkokkoscontainers -lkokkoscore -ltpi -lgtest /usr/lib/x86_64-linux-gnu/libX11.so /usr/lib/x86_64-linux-gnu/libboost_program_options.so /usr/lib/x86_64-linux-gnu/libboost_system.so /home/mhoffman/software/netcdf/netcdf-4.3.2-install/lib/libnetcdf.so /usr/lib/x86_64-linux-gnu/libhdf5.so /usr/lib/x86_64-linux-gnu/libz.so /usr/lib/liblapack.so /usr/lib/libblas.so -L/usr/lib -lmpi_cxx -lmpi_f90 -lstdc++

 

it is built like this:

export MPAS_EXTERNAL_LIBS="$ALBANY_LIBS $Trilinos_LIBRARY_DIRS $Trilinos_LIBRARIES $Trilinos_TPL_LIBRARIES -L/usr/lib -lmpi_cxx -lmpi_f90 -lstdc++"

where ALBANY_LIBS is manually-maintained based on Albany development changes, but currently looks like:

export ALBANY_LIBS="-L/home/mhoffman/software/albany/albany-install-mpas/lib -lmpasInterface -lalbanyLib -lalbanySTK -lFELIX -lalbanyAdapt -lalbanySTKRebalance"

and the Trilinos-related libraries are defined in the "Makefile.export.Trilinos_install" file created when building Trilinos.  Currently it looks something like this:
## The project library directories.
export Trilinos_LIBRARY_DIRS="-L/home/mhoffman/software/trilinos/trilinos-git-install/lib"
## The project libraries.
export Trilinos_LIBRARIES="-lpiro -lstokhos_muelu -lstokhos_ifpack2 -lstokhos_amesos2 -lstokhos_tpetra -lstokhos_sacado -lstokhos -lrythmos -lmuelu-adapters -lmuelu-interface -lmuelu -llocathyra -llocaepetra -llocalapack -lloca -lnoxepetra -lnoxlapack -lnox -lphalanx -lstk_unit_test_utils -lstk_search -lstk_io_util -lstk_io -lstk_mesh_base -lstk_topology -lstk_util_use_cases -lstk_util_registry -lstk_util_parallel -lstk_util_diag -lstk_util_env -lstk_util_util -lstk_unit_test_utils -lstk_search -lstk_io_util -lstk_io -lstk_mesh_base -lstk_topology -lstk_util_use_cases -lstk_util_registry -lstk_util_parallel -lstk_util_diag -lstk_util_env -lstk_util_util -lintrepid -lteko -lstratimikos -lstratimikosbelos -lstratimikosaztecoo -lstratimikosamesos -lstratimikosml -lstratimikosifpack -lifpack2-adapters -lifpack2 -lzoltan2 -lanasazitpetra -lModeLaplace -lanasaziepetra -lanasazi -lbelostpetra -lbelosepetra -lbelos -lml -lifpack -lmapvarlib -lfastqlib -lblotlib -lplt -lsvdi_cgi -lsvdi_cdr -lsuplib -lsupes -laprepro_lib -lchaco -lIonit -lIotr -lIohb -lIogn -lIopg -lIoexo_fac -lIopx -lIofx -lIoex -lIoss -lnemesis -lexodus_for -lexodus -lmapvarlib -lfastqlib -lblotlib -lplt -lsvdi_cgi -lsvdi_cdr -lsuplib -lsupes -laprepro_lib -lchaco -lIonit -lIotr -lIohb -lIogn -lIopg -lIoexo_fac -lIopx -lIofx -lIoex -lIoss -lnemesis -lexodus_for -lexodus -lpamgen_extras -lpamgen -lamesos2 -lamesos -lgaleri-xpetra -lgaleri -laztecoo -lisorropia -loptipack -lthyratpetra -lthyraepetraext -lthyraepetra -lthyracore -lthyratpetra -lthyraepetraext -lthyraepetra -lthyracore -lxpetra-sup -lxpetra-ext -lxpetra -lepetraext -ltpetraext -ltpetrainout -ltpetra -lkokkostsqr -ltpetrakernels -ltpetraclassiclinalg -ltpetraclassicnodeapi -ltpetraclassic -ltpetraext -ltpetrainout -ltpetra -lkokkostsqr -ltpetrakernels -ltpetraclassiclinalg -ltpetraclassicnodeapi -ltpetraclassic -ltriutils -lglobipack -lshards -lzoltan -lepetra -lsacado -lrtop -lteuchoskokkoscomm -lteuchoskokkoscompat -lteuchosremainder -lteuchosnumerics -lteuchoscomm -lteuchosparameterlist -lteuchoscore -lteuchoskokkoscomm -lteuchoskokkoscompat -lteuchosremainder -lteuchosnumerics -lteuchoscomm -lteuchosparameterlist -lteuchoscore -lkokkosalgorithms -lkokkoscontainers -lkokkoscore -lkokkosalgorithms -lkokkoscontainers -lkokkoscore -ltpi -lgtest"
## The project tpl libraries
export Trilinos_TPL_LIBRARIES="/usr/lib/x86_64-linux-gnu/libX11.so /usr/lib/x86_64-linux-gnu/libboost_program_options.so /usr/lib/x86_64-linux-gnu/libboost_system.so /home/mhoffman/software/netcdf/netcdf-4.3.2-install/lib/libnetcdf.so /usr/lib/x86_64-linux-gnu/libhdf5.so /usr/lib/x86_64-linux-gnu/libz.so /usr/lib/liblapack.so /usr/lib/libblas.so"

In standalone MPASLI, we typically have created environment variables to store this information, so that within a user's environment they simply need to compile MPAS with, e.g., "make gfortran ALBANY=true" and the MPAS_EXTERNAL_LIBS environment variable is automatically included in the build.  However, in ACME it appears that a user's environment is ignored when building ACME, so this approach may not be possible.  One alternative is to specify this information in the machine files for every machine that Albany is supported.  However, I fear this could be hard to maintain.  Also, for non-specific machine files, like linux-generic and mac, it's not clear to me how to implement this in a user-friendly way that does not require the user to manually edit the machine file (or do something complicated like set up modules on a machine that otherwise does not use them).
Andy Salinger, do you have thoughts about how to deal with this?  

Jon Wolfe suggested we look into how Trilinos is currently supported in CESM/ACME.

Note that this gets a little easier to manage if modules for Trilinos and Albany are created, but that does not eliminate the fundamental issue about how to get this environment variable into ACME.  I have created Trilinos and Albany modules for gcc/openmpi on Mustang and Wolf.  These modules set the environment variables required.


Planned Verification and Unit Testing 

Verification and Unit Testing: Verify Working Builds on LCF (and other testing) Machines

Date last modified: 2015/9/4

Contributors: Matt Hoffman


I have implemented and tested items 1 and 2, and they appear to be working. Note I had to remove the dash from the MPAS-LI/mpas-li component name to get the *_CONFIG_OPTS variable to work correctly.  Details of that issue are described here: /wiki/spaces/OCNICE/pages/18939976
But the build dies because MPAS_EXTERNAL_LIBS is not set, so item 3 remains unresolved.
Matt Hoffman and Andy Salinger - I think getting the builds to work would qualify as a "verification" step. Can you add a few sentences to describe the status for each platform listed below?

Status on LANL HPC:

 

Status on Edison:

 

Status on Titan:

 

Status on Mira:


Planned Validation Testing 


Validation testing is not applicable for this task. Verification testing - i.e. that the changes to the build system needed for MPAS-LI in ACME to call the third-party libraries - should be adequate.

Planned Performance Testing


Performance Testing:

Date last modified: 2015/9/22
Contributors: Stephen Price

 

This is part of a new land ice model component so there are no previous benchmarks for model performance testing. See the link to the overal MPAS-Land Ice model integration design document for additional discussion on performance.







  • No labels