E3SM IDP Team
Date:
Summary
This design describes the deploy subpackage of the mache package that can be used to create conda and spack environments on E3SM-supported machines for E3SM-supported software, including Compass, Polaris, and E3SM-Unified. The motivation for this new package is to unify redundant and poorly documented code for environment deployment in these 3 packages, to better document it, and to provide an approach that could be adopted by future E3SM software that has both conda and spack dependencies.
Requirements
Date last modified:
Contributors: Xylar Asay-Davis Althea Denlinger
In what follows, we will refer to the software such as Polaris or E3SM-Unified that will use mache.deploy as the “deployed software”.
Requirement: A mechanism to begin deployment
The deployed software needs a way to begin the deployment process. This cannot be provided directly by mache because it will not yet be installed at this point, but mache.deploy or its documentation should provide a suggested process for the deployed software to adopt.
Requirement: A mechanism to install miniforge3
The deployed software needs a way to install miniforge3 if a developer doesn’t already have it installed. This cannot be provided directly by mache because it will not yet be installed at this point.
Requirement: A mechanism to install mache
The deployed software needs a way to install mache from either conda-forge or a user-specified branch. Again, this cannot be provided directly by mache because it will not yet be installed at this point.
Requirement: A way for deployed software to specify conda packages to install
The mache.deploy package must be able create a conda environment with a list of dependencies including version constraints specified by the deployed software.
Requirement: A way for deployed software to specify spack packages to install
The mache.deploy package must be able create a spack environment with a list of dependencies including version and variant constraints specified by the deployed software.
Requirement: Support for different variants of the environments
The mache.deploy package should support a mechanism for the deployed software to provide different variants of the conda and spack environments. These variants might be for different machines, compilers and/or MPI variants, but they also may include or exclude specific packages (e.g. Trilinos, Albany or PETSc). There should be a clear way that the supporting software can define these variants, and a way that a developer can request a specific variant (or a list of variants) when deploying.
Requirement: Include a mechanism for providing environment variables
The deployed software may need to specify environment variables to be set before it can run (e.g. paths to libraries needed to build standalone E3SM components). There should be a way to specify these and for them to be included in the “load” mechanism described below.
Requirement: Provide a mechanism to load the environments in the deployed software
The mache.deploy package must provide a mechanism for loading the conda and spack environments, loading system modules, and setting environment variables before the deployed software gets run.
Requirement: Skip spack deployment
The deployed software must have a way to indicate that spack deployment isn’t needed, either because a shared spack environment already exists or because no spack support is needed.
Desired: Skip conda deployment
The deployed software may wish to have a way to indicate that conda deployment isn’t needed because a shared conda environment already exists. Such a capability is not currently needed but it could be in future software. Deployment would still be useful in such cases because mache.deploy would still provide the load mechanism described above.
Desired: Testing the deployed software
It would be desirable if mache.deploy includes a mechanism for the deployed software to provide tests that can be run to ensure that deployment was successful.
Conceptual Design
Date last modified:
Contributors: Xylar Asay-Davis Althea Denlinger
Design: A mechanism to begin deployment
A generalized configure script will install mache and then carry out deployment similarly to the configure script and deployment package in polaris. This script will be retrieved from github with a short localized script in the software to be deployed (e.g., polaris would have a small localized script for retrieving the script to install conda, mache, and begin deployment). So there will be three scripts: 1.) localized script within the target software (e.g., polaris), 2.) begin deployment script which will be used to install conda and mache, and 3.) a script like the configure_polaris_envs.py script which will complete the rest of the deployment process.
Design: A mechanism to install miniforge3
Included in the mache deployment script.
Design: A mechanism to install mache
Included in the mache deployment script.
Design: A way for deployed software to specify conda packages to install
The mache documentation will describe how to set up a spec file that will describe which conda packages to install.
Design: A way for deployed software to specify spack packages to install
The mache documentation will describe how to set up a spec file that will describe which spack packages to install.