E3SM IDP Team
Date:
Summary
This design describes the deploy
subpackage of the mache
package that can be used to create conda and spack environments on E3SM-supported machines for E3SM-supported software, including Compass, Polaris, and E3SM-Unified. The motivation for this new package is to unify redundant and poorly documented code for environment deployment in these 3 packages, to better document it, and to provide an approach that could be adopted by future E3SM software that has both conda and spack dependencies.
Requirements
Date last modified:
Contributors: Xylar Asay-Davis Althea Denlinger
In what follows, we will refer to the software such as Polaris or E3SM-Unified that will use mache.deploy
as the “deployed software”.
Requirement: A mechanism to begin deployment
The deployed software needs a way to begin the deployment process. This cannot be provided directly by mache
because it will not yet be installed at this point, but mache.deploy
or its documentation should provide a suggested process for the deployed software to adopt.
Requirement: A mechanism to install miniforge3
The deployed software needs a way to install miniforge3
if a developer doesn’t already have it installed. This cannot be provided directly by mache
because it will not yet be installed at this point.
Requirement: A mechanism to install mache
The deployed software needs a way to install mache
from either conda-forge or a user-specified branch. Again, this cannot be provided directly by mache
because it will not yet be installed at this point.
Requirement: A way for deployed software to specify conda packages to install
The mache.deploy
package must be able create a conda environment with a list of dependencies including version constraints specified by the deployed software.
Requirement: A way for deployed software to specify spack packages to install
The mache.deploy
package must be able create a spack environment with a list of dependencies including version and variant constraints specified by the deployed software.
Requirement: Support for different variants of the environments
The mache.deploy
package should support a mechanism for the deployed software to provide different variants of the conda and spack environments. These variants might be for different machines, compilers and/or MPI variants, but they also may include or exclude specific packages (e.g. Trilinos, Albany or PETSc). There should be a clear way that the supporting software can define these variants, and a way that a developer can request a specific variant (or a list of variants) when deploying.
Requirement: Include a mechanism for providing environment variables
The deployed software may need to specify environment variables to be set before it can run (e.g. paths to libraries needed to build standalone E3SM components). There should be a way to specify these and for them to be included in the “load” mechanism described below.
Requirement: Provide a mechanism to load the environments in the deployed software
The mache.deploy
package must provide a mechanism for loading the conda and spack environments, loading system modules, and setting environment variables before the deployed software gets run.
Desired: Testing the deployed software
It would be desirable if mache.deploy
includes a mechanism for the deployed software to provide tests that can be run to ensure that deployment was successful.