E3SM IDP Team
Date:
Summary
A python package that creates and deploys This design describes the deploy
subpackage of the mache
package that can be used to create conda and spack environments on E3SM-supported machines for E3SM-supported software, including Compass, Polaris, and E3SM-Unified. The motivation for this new package is to unify redundant and poorly documented code for environment deployment in these 3 packages, to better document it, and to provide an approach that could be adopted by future E3SM software that has both conda and spack dependencies.
Requirements
Date last modified:
Contributors: Xylar Asay-Davis Althea Denlinger
Requirement:
Create a bootstrap conda environment with which In what follows, we will refer to the software such as Polaris or E3SM-Unified that will use mache.deploy
as the “deployed software”.
Requirement: A mechanism to begin deployment
The deployed software needs a way to begin the deployment process. This cannot be provided directly by mache
because it will not yet be installed at this point, but mache.deploy
or its documentation should provide a suggested process for the deployed software to adopt.
Requirement: A mechanism to install miniforge3
The deployed software needs a way to install miniforge3
if a developer doesn’t already have it installed. This cannot be provided directly by mache
because it will not yet be installed at this point.
Requirement: A mechanism to install mache
The deployed software needs a way to install mache
from either conda-forge or a given branch.
...
user-specified branch. Again, this cannot be provided directly by mache
because it will not yet be installed at this point.
Requirement: A way for deployed software to specify conda packages to install
The mache.deploy
package must be able create a conda environment with a list of dependencies including version constraints specified by the deployed software.
Requirement: A way for deployed software to specify spack packages to install
The mache.deploy
package must be able create a spack environment with a list of dependencies including version and variant constraints specified by the deployed software.
Requirement: Support for different variants of the environments
The mache.deploy
package should support a mechanism for the deployed software to provide different variants of the conda and spack environments. These variants might be for different machines, compilers and/or MPI variants, but they also may include or exclude specific packages (e.g. Trilinos, Albany or PETSc). There should be a clear way that the supporting software can define these variants, and a way that a developer can request a specific variant (or a list of variants) when deploying.
Requirement: Include a mechanism for providing environment variables
The deployed software may need to specify environment variables to be set before it can run (e.g. paths to libraries needed to build standalone E3SM components). There should be a way to specify these and for them to be included in the “load” mechanism described below.
Requirement: Provide a mechanism to load the environments in the deployed software
The mache.deploy
package must provide a mechanism for loading the conda and spack environments, loading system modules, and setting environment variables before the deployed software gets run.
Requirement: Skip spack deployment
The deployed software must have a way to indicate that spack deployment isn’t needed, either because a shared spack environment already exists or because no spack support is needed.
Desired: Skip conda deployment
The deployed software may wish to have a way to indicate that conda deployment isn’t needed because a shared conda environment already exists. Such a capability is not currently needed but it could be in future software. Deployment would still be useful in such cases because mache.deploy
would still provide the load mechanism described above.
Desired: Testing the deployed software
It would be desirable if mache.deploy
includes a mechanism for the deployed software to provide tests that can be run to ensure that deployment was successful.
Conceptual Design
Date last modified:
Contributors: Xylar Asay-Davis Althea Denlinger
Algorithm:
...
Design: A mechanism to begin deployment
A generalized configure script will install mache and then carry out deployment similarly to the configure script and deployment package in polaris. This script will be retrieved from github with a short localized script in the software to be deployed (e.g., polaris would have a small localized script for retrieving the script to install conda, mache, and begin deployment). So there will be three scripts: 1.) localized script within the target software (e.g., polaris), 2.) begin deployment script which will be used to install conda and mache, and 3.) a script like the configure_polaris_envs.py script which will complete the rest of the deployment process.
Design: A mechanism to install miniforge3
Included in the mache deployment script.
Design: A mechanism to install mache
Included in the mache deployment script.
Design: A way for deployed software to specify conda packages to install
The mache documentation will describe how to set up a spec file that will describe which conda packages to install.
Design: A way for deployed software to specify spack packages to install
The mache documentation will describe how to set up a spec file that will describe which spack packages to install.
Design: Support for different variants of the environments
Design: Include a mechanism for providing environment variables
Design: Provide a mechanism to load the environments in the deployed software
Design: Skip spack deployment
Optional Design: Skip conda deployment
Optional Design: Testing the deployed software
Pseudo-code and Implementation Details
Date last modified:
Contributors:
...
Xylar Asay-Davis Althea Denlinger
Implementation: A mechanism to begin deployment
Implementation: A mechanism to install miniforge3
Implementation: A mechanism to install mache
Implementation: A way for deployed software to specify conda packages to install
Implementation: A way for deployed software to specify spack packages to install
Implementation: Support for different variants of the environments
Implementation: Include a mechanism for providing environment variables
Implementation: Provide a mechanism to load the environments in the deployed software
Implementation: Skip spack deployment
Optional Implementation: Skip conda deployment
Optional Implementation: Testing the deployed software
Planned Testing
Date last modified:
...