THIS PAGE HAS BEEN REPLACED BY COMPASS READ-THE-DOCS. PLEASE GO THERE.

Information on this page is no longer updated.

Overview

...

Configuration Of Model for Prediction Across Scales Setups (COMPASS) is an automated system to set up test cases that match the MPAS-Ocean repository. All namelists and streams files begin with the default generated from the Registry.xml file, and only the changes relevant to the particular test case are altered in those files.

...

Set up a COMPASS repository: for beginners

To begin, obtain the master branch of the compass repository with:

git clone git@github.com:MPAS-Dev/compass.git
cd compass
git submodule update --init --recursive

The MPAS repository is a submodule of COMPASS repository. Load the compass conda package for your particular machine. For example, to compile MPAS-Ocean:

# on LANL IC:
source /usr/projects/climate/SHARED_CLIMATE/anaconda_envs/load_latest_compass.sh
# at NERSC:
source /global/cd MPAS-Model/ocean/develop/
# load modules (see machine-specific instructions below)
make gfortran CORE=ocean

Load the compass conda environment

The compass conda environment includes all the python libraries needed to run compass scripts. It is maintained on our standard machines:

# on LANL IC:
source /usr/projects/climate/SHARED_CLIMATE/anaconda_envs/load_latest_compass.sh
# at NERSC:
source /global/cfs/cdirs/e3sm/software/anaconda_envs/load_latest_compass.sh

To install your own compass environment, see https://github.com/MPAS-Dev/MPAS-Model/blob/ocean/develop/testing_and_setup/compass/README_ocean.md You will also need to git clone an MPAS repo. It is usually best to start in the ocean/develop branch. on other machines:

# Do this once:
conda create -n compass_0.1.11 -c conda-forge -c e3sm python=3.7 compass=0.1.11
# Do this every time you need to load the compass environment
conda activate compass_0.1.1

Setting up a test case

If you are new to MPAS-Ocean, it is easiest to download a prepared test case. To see all available test cases you can make yourself in compass, start in the ocean/develop branch.:

...

./setup_testcase.py \
  --config_file general.config.ocean \
  --work_dir $WORKDIR \
  --model_runtime runtime_definitions/mpirun.xml \
  -o ocean -c global_ocean -r QU240 -t init

or you can replace the last line with the simple shortcut: -n 69.

Here $WORKDIR is a path, usually to your scratch space. For example,

--work_dir /lustre/scratch4/turquoise/mpeterse/runs/191210_test_new_branch

and general.config.ocean is a file that specifies directory and file paths. You can either add paths to the repo file in that directory, or you can use these files, which use my paths:

• LANL IC: general.config.ocean_turq

• NERSC: general.config.ocean_cori

You should change the MPAS repo to your directories to test your own code.

The --model_runtime is either srun or mpirun, depending which modules you loaded.

Running a test case

After compiling the code and setting up a test case, you can log into an interactive node (see machine instructions below) and then

cd $WORKDIR
cd ocean/global_ocean/QU240/init
./run.py

Note the sequence of subdirectories is the same as the flags used to set up the case.

In order to run a bit-for-bit test with a previous case, use -b $PREVIOUS_WORKDIR.

Regression suites

We have assembles suites of test cases for code regressions and bit-for-bit testing. They are here:

ls testing_and_setup/compass/ocean/regression_suites/

land_ice_fluxes.xml light.xml nightly.xml rpe_tests.xml

You can set up a regression as follows:

cd testing_and_setup/compass/
./manage_regression_suite.py -s \
   --config_file general.config.ocean \
   -t ocean/regression_suites/nightly.xml \
   --model_runtime runtime_definitions/mpirun.xml \
   --work_dir $WORKDIR

where the details are the same as for setting up a case. You can use the same general.config.ocean file and use -b $PREVIOUS_WORKDIR for bit-for-bit comparison of the output with a previous nightly regression suite.

To run the regression suite, log into an interactive node, load your modules, and

cd $WORKDIR
./nightly_ocean_test_suite.py

Machine-Specific Instructions

...

.xml \
  -o ocean -c global_ocean -r QU240 -t init

or you can replace the last line with the simple shortcut: -n 69.

Here $WORKDIR is a path, usually to your scratch space. For example,

--work_dir /lustre/scratch4/turquoise/$USER/runs/191210_test_new_branch

and general.config.ocean is a file that specifies directory and file paths. You can either add paths to the repo file in that directory, or you can use these files, which use my paths:

• LANL IC: config.ocean_LANL_IC

• NERSC: general.config.ocean_cori

You should change the MPAS repo to your directories to test your own code.

The --model_runtime is either srun or mpirun, depending which modules you loaded.

Running a test case

After compiling the code and setting up a test case, you can log into an interactive node (see machine instructions below) and then

cd $WORKDIR
cd ocean/global_ocean/QU240/init
./run.py

Note the sequence of subdirectories is the same as the flags used to set up the case.

In order to run a bit-for-bit test with a previous case, use -b $PREVIOUS_WORKDIR.

Regression suites

We have assembles suites of test cases for code regressions and bit-for-bit testing. They are here:

ls testing_and_setup/compass/ocean/regression_suites/

land_ice_fluxes.xml light.xml nightly.xml rpe_tests.xml

You can set up a regression as follows:

cd testing_and_setup/compass/
./manage_regression_suite.py -s \
   --config_file general.config.ocean \
   -t ocean/regression_suites/nightly.xml \
   --model_runtime runtime_definitions/mpirun.xml \
   --work_dir $WORKDIR

where the details are the same as for setting up a case. You can use the same general.config.ocean file and use -b $PREVIOUS_WORKDIR for bit-for-bit comparison of the output with a previous nightly regression suite.

To run the regression suite, log into an interactive node, load your modules, and

cd $WORKDIR
./nightly_ocean_test_suite.py

Set up a COMPASS repository with worktrees: for advanced users

This section uses git worktree, which provides more flexibility but is more complicated. See the beginner section above for the simpler version. In the worktree version, you will have many unix directories, and each corresponds to a git branch. It is easier to keep track of, and easier to work with many branches at once. Begin where you keep your repositories:

mkdir compass
cd compass
git clone git@github.com:MPAS-Dev/compass.git master
cd master

The MPAS-Dev/compass is now origin. You can add more remotes. For example

git remote add mark-petersen git@github.com:mark-petersen/compass.git
git fetch mark-petersen

To view all your remotes:

git remote -v

To view all available branches, both local and remote:

git branch -a

We will use the git worktree command to create a new local branch in its own unix directory.

cd compass/master
git worktree add -b newBranchName ../newBranchName origin/master
cd ../newBranchName

In this example, we branched off origin/master, but you could start from any branch, which is specified by the last git worktree argument.

In each new branch directory that you make, you will need to alter the general.config.ocean file to point to the MPAS executable and files. There are two ways to point to the MPAS executable:

1. Compass submodule (easier): This guarantees that the MPAS commit matches compass.

   Code Block
   git submodule update --init --recursive cd MPAS-Model/ocean/develop/ # load modules (see machine-specific instructions below) make gfortran CORE=ocean

2. Other MPAS directory (advanced): Create your own MPAS repository elsewhere on disk, and point general.config.ocean to that. The user must ensure that flag names and test cases match appropriately. The simplest way to set up a new MPAS repo in a new directory is:

   Code Block
   git clone git@github.com:MPAS-Dev/MPAS.git your_new_branch cd your_new_branch git checkout -b your_new_branch origin/ocean/develop

   Note that for ocean development, it is best to branch from ocean/develop.

...

Machine-Specific Instructions

Slurm job queueing

Most systems now use slurm. Here are some basic commands:

...

• NERSC uses HPSS with the commands hsi and htar

• overview: https://docs.nersc.gov/filesystems/archive/

• E3SM uses zstash B04. zstash: HPSS long-term archiving tool

cori, gnu

module switch PrgEnv-intel PrgEnv-gnu
module load cray-netcdf-hdf5parallel
module load cray-parallel-netcdf
module load cmake
source /global/project/projectdirs/e3sm/software/anaconda_envs/load_latest_e3sm_unified.sh
export PIO=/global/u2/h/hgkang/my_programs/Scorpio
git submodule update --init --recursive

# debug:
make gnu-nersc CORE=ocean USE_PIO2=true OPENMP=false DEBUG=true GEN_F90=true

# optimized:
make gnu-nersc CORE=ocean USE_PIO2=true OPENMP=false

...