...
sudo singularity build e3sm.sif e3sm.def
It may take up to an hour to create a new container.
Run the E3SM developer tests
...
git clone -b maint-1.2 --recursive git@github.com:E3SM-Project/E3SM.git
Modify Add a corresponding machine element to cime/config/e3sm/machines/config_machines.xml, so a value of the NODENAME_REGEX element of the linux-generic machine is singularity. Remove openmpi from MPILIBS:
| Code Block | ||
|---|---|---|
| ||
<MPILIBS>mpich</MPILIBS> |
Specify GMAKE command above the GMAKE_J element:
| Code Block |
|---|
<GMAKE>make</GMAKE>
<GMAKE_J>4</GMAKE_J> |
Define also environment variables:
| Code Block | ||
|---|---|---|
| ||
<environment_variables><machine MACH="singularity"> <DESC>Singularity container</DESC> <NODENAME_REGEX>singularity</NODENAME_REGEX> <OS>LINUX</OS> <COMPILERS>gnu</COMPILERS> <MPILIBS>mpich</MPILIBS> <CIME_OUTPUT_ROOT>$ENV{HOME}/projects/acme/scratch</CIME_OUTPUT_ROOT> <DIN_LOC_ROOT>$ENV{HOME}/projects/acme/cesm-inputdata</DIN_LOC_ROOT> <DIN_LOC_ROOT_CLMFORC>$ENV{HOME}/projects/acme/ptclm-data</DIN_LOC_ROOT_CLMFORC> <DOUT_S_ROOT>$ENV{HOME}/projects/acme/scratch/archive/$CASE</DOUT_S_ROOT> <BASELINE_ROOT>$ENV{HOME}/projects/acme/baselines/$COMPILER</BASELINE_ROOT> <CCSM_CPRNC>$CCSMROOT/tools/cprnc/build/cprnc</CCSM_CPRNC> <GMAKE>make</GMAKE> <GMAKE_J>16</GMAKE_J> <TESTS>e3sm_developer</TESTS> <BATCH_SYSTEM>none</BATCH_SYSTEM> <SUPPORTED_BY>jayesh at mcs dot anl dot gov</SUPPORTED_BY> <MAX_TASKS_PER_NODE>4</MAX_TASKS_PER_NODE> <MAX_MPITASKS_PER_NODE>2</MAX_MPITASKS_PER_NODE> <mpirun mpilib="default"> <executable>mpirun</executable> <arguments> <arg name="num_tasks"> -np {{ total_tasks }}</arg> </arguments> </mpirun> <module_system type="none"/> <RUNDIR>$ENV{HOME}/projects/acme/scratch/$CASE/run</RUNDIR> <EXEROOT>$ENV{HOME}/projects/acme/scratch/$CASE/bld</EXEROOT> <!-- cmake -DCMAKE_Fortran_COMPILER=/opt/local/bin/mpif90-mpich-gcc48 -DHDF5_DIR=/opt/local -DNetcdf_INCLUDE_DIR=/opt/local/include .. --> <!-- <GMAKE>make</GMAKE> <- this doesn't actually work! --> <environment_variables> <env name="E3SM_SRCROOT">$SRCROOT</env> </environment_variables> <environment_variables mpilib="mpi-serial"> <env name="NETCDF_PATH">/usr/local/packages/netcdf-serial</env> <env name="PATH">/usr/local/packages/hdf5-1.10.6-serial/bin:/usr/local/packages/netcdf-serial/bin:$ENV{PATH}</env> <env name="LD_LIBRARY_PATH">/usr/local/packages/szip-2.1.1/lib:/usr/local/packages/hdf5-1.10.6-serial/lib:/usr/local/packages/netcdf-serial/lib</env> </environment_variables> <environment_variables mpilib="!mpi-serial"> <env name="NETCDF_PATH">/usr/local/packages/netcdf-parallel</env> <env name="PNETCDF_PATH">/usr/local/packages/pnetcdf-1.12.1</env> <env name="PATH">/usr/local/packages/mpich-3.3.2/bin:/usr/local/packages/hdf5-1.10.6-parallel/bin:/usr/local/packages/netcdf-parallel/bin:/usr/local/packages/pnetcdf-1.12.1/bin:$ENV{PATH}</env> <env name="LD_LIBRARY_PATH">/usr/local/packages/mpich-3.3.2/lib:/usr/local/packages/szip-2.1.1/lib:/usr/local/packages/hdf5-1.10.6-parallel/lib:/usr/local/packages/netcdf-parallel/lib:/usr/local/packages/pnetcdf-1.12.1/lib</env> </environment_variables> </machine> |
Then add options -lblas -llapack to a linker in a corresponding compiler element to cime/config/e3sm/machines/config_compilers.xml:
| Code Block | ||
|---|---|---|
| ||
<compiler COMPILER="gnu" MACH="linux-genericsingularity"> <NETCDF_PATH> $(NETCDF_PATH)</NETCDF_PATH> <PNETCDF_PATH> $(PNETCDF_PATH)</PNETCDF_PATH> <ADD_SLIBS> $(shell $(NETCDF_PATH)/bin/nf-config --flibs) -lblas -llapack</ADD_SLIBS> </compiler> |
...
mkdir $HOME/projects
singularity shell --hostname singularity -B $HOME/projects e3sm.sif
...