THIS PAGE HAS BEEN REPLACED BY COMPASS READ-THE-DOCS. PLEASE GO THERE.

Information on this page is no longer updated.

Overview

1. MPAS website: https://mpas-dev.github.io/

2. MPAS-Ocean User's Guide includes a quick start guide and description of all flags and variables.

3. Test cases for latest release version:

   1. https://mpas-dev.github.io/ocean/releases.html

   2. https://mpas-dev.github.io/ocean/release_6.0/release_6.0.html

COMPASS

Configuration Of Model for Prediction Across Scales Setups (COMPASS) is an automated system to set up test cases that match the MPAS-Ocean repository. All namelists and streams files begin with the default generated from the Registry.xml file, and only the changes relevant to the particular test case are altered in those files.

Set up a COMPASS repository: for beginners

To begin, obtain the master branch of the compass repository with:

The MPAS repository is a submodule of COMPASS repository. For example, to compile MPAS-Ocean:

Load the compass conda environment

The compass conda environment includes all the python libraries needed to run compass scripts. It is maintained on our standard machines:

To install your own compass environment on other machines:

Setting up a test case

If you are new to MPAS-Ocean, it is easiest to download a prepared test case. To see all available test cases you can make yourself in compass, start in the ocean/develop branch.:

cd testing_and_setup/compass

./list_testcases.py

and you get output like this:

69: -o ocean -c global_ocean -r QU240 -t init

70: -o ocean -c global_ocean -r QU240 -t performance_test

To set up a particular test case, you can either use the full sequence of flags:

or you can replace the last line with the simple shortcut: -n 69.

Here $WORKDIR is a path, usually to your scratch space. For example,

--work_dir /lustre/scratch4/turquoise/$USER/runs/191210_test_new_branch

and general.config.ocean is a file that specifies directory and file paths. You can either add paths to the repo file in that directory, or you can use these files, which use my paths:

• LANL IC: config.ocean_LANL_IC

• NERSC: general.config.ocean_cori

You should change the MPAS repo to your directories to test your own code.

The --model_runtime is either srun or mpirun, depending which modules you loaded.

Running a test case

After compiling the code and setting up a test case, you can log into an interactive node (see machine instructions below) and then

Note the sequence of subdirectories is the same as the flags used to set up the case.

In order to run a bit-for-bit test with a previous case, use -b $PREVIOUS_WORKDIR.

Regression suites

We have assembles suites of test cases for code regressions and bit-for-bit testing. They are here:

ls testing_and_setup/compass/ocean/regression_suites/

land_ice_fluxes.xml light.xml nightly.xml rpe_tests.xml

You can set up a regression as follows:

where the details are the same as for setting up a case. You can use the same general.config.ocean file and use -b $PREVIOUS_WORKDIR for bit-for-bit comparison of the output with a previous nightly regression suite.

To run the regression suite, log into an interactive node, load your modules, and

Set up a COMPASS repository with worktrees: for advanced users

This section uses git worktree, which provides more flexibility but is more complicated. See the beginner section above for the simpler version. In the worktree version, you will have many unix directories, and each corresponds to a git branch. It is easier to keep track of, and easier to work with many branches at once. Begin where you keep your repositories:

The MPAS-Dev/compass is now origin. You can add more remotes. For example

To view all your remotes:

To view all available branches, both local and remote:

We will use the git worktree command to create a new local branch in its own unix directory.

In this example, we branched off origin/master, but you could start from any branch, which is specified by the last git worktree argument.

In each new branch directory that you make, you will need to alter the general.config.ocean file to point to the MPAS executable and files. There are two ways to point to the MPAS executable:

1. Compass submodule (easier): This guarantees that the MPAS commit matches compass.

2. Other MPAS directory (advanced): Create your own MPAS repository elsewhere on disk, and point general.config.ocean to that. The user must ensure that flag names and test cases match appropriately. The simplest way to set up a new MPAS repo in a new directory is:

   Note that for ocean development, it is best to branch from ocean/develop.

Machine-Specific Instructions

Slurm job queueing

Most systems now use slurm. Here are some basic commands:

Also see:

• Introduction to Slurm at LANL

• Basic Slurm Guide for LANL HPC Users

• Slurm Command Summary

• Slurm: Running Jobs on HPC Platforms

• example of batch scripts: https://hpc.lanl.gov/slurm#batch_scripts

LANL Institutional Computing

See https://int.lanl.gov/hpc/institutional-computing/index.shtml

DST Calendar: http://hpccalendar.lanl.gov/

Machine specifications: grizzly badger turquoise network

login: ssh -t $my_moniker@wtrw.lanl.gov ssh gr-fe or ba-fe

File locations:

• small home directory, for start-up scripts only: /users/$my_moniker

• home directory, backed up: /usr/projects/climate/$my_moniker

• scratch space, not backed up: /lustre/scratch3/turquoise/$my_moniker or scratch4

Check compute time:

• sacctmgr list assoc user=$my_moniker format=Cluster,Account%18,Partition,QOS%45

• Which is my default account? sacctmgr list user $my_moniker

• sshare -a | head -2; sshare -a | grep $ACCOUNT | head -1

• sreport -t Hours cluster AccountUtilizationByUser start=2019-12-02 | grep $ACCOUNT

• check job priority: sshare -a | head -2; sshare -a | grep $ACCOUNT

• https://hpcinfo.lanl.gov

• https://hpcstats.lanl.gov

Check disk usage:

• your home space: chkhome

• total disk usage in Petabytes: df -BP |head -n 1; df -BP|grep climate; df -BP |grep scratch

Archiving

• see https://hpc.lanl.gov/turquoise_archive

• archive front end: ssh -t $my_moniker@wtrw.lanl.gov ssh ar-tn

• storage available at: cd /archive/<project_name>

• you can just copy files directly into here for a particular project.

git and compass environment, for all LANL IC machines:

Example compass config file for LANL IC: general.config.ocean_turq

LANL uses slurm. To obtain an interactive node:

use --account=ACCOUNT_NAME to change to a particular account.

grizzly, gnu

Hint: you can put the following line in your bashrc:

alias mlgnu='module purge; module load git; module use /usr/projects/climate/SHARED_CLIMATE/modulefiles/all/; module load gcc/5.3.0 openmpi/1.10.5 netcdf/4.4.1 parallel-netcdf/1.5.0 pio/1.7.2; module unload python; source /usr/projects/climate/SHARED_CLIMATE/anaconda_envs/load_latest_compass.sh; echo "loading modules anaconda, gnu, openmpi, netcdf, pnetcdf, pio for grizzly"

grizzly, intel 17

grizzly, intel 19 with PIO2

badger, gnu

NERSC

login: ssh $my_moniker@cori.nersc.gov

compass environment:

example compass config file: general.config.ocean_cori

interactive login:

Compute time:

• Check hours of compute usage at https://nim.nersc.gov/

File system:

• Overview: https://docs.nersc.gov/filesystems/

• home directory: /global/homes/$my_moniker

• scratch directory: /global/cscratch1/sd/$my_moniker

• Check your individual disk usage with myquota

• Check the group disk usage with prjquota projectID, i.e. prjquota m2833 or prjquota acme

Archive:

• NERSC uses HPSS with the commands hsi and htar

• overview: https://docs.nersc.gov/filesystems/archive/

• E3SM uses zstash B04. zstash: HPSS long-term archiving tool

cori, gnu

cori, intel

PIO on cori

We have already compiled PIO on cori, and paths are given in the previous instructions. If you need to compile it yourself, you can do that as follows (thanks @Xylar Asay-Davis for instructions).

Jupyter notebook on remote data

You can run Jupyter notebooks on NERSC with direct access to scratch data as follows:

Note that on NERSC, you can also use their Jupyter server (Jupyter.nersc.gov), it’s really nice and grabs a compute node for you automatically on logon. You’ll need to create a python kernel from e3sm-unified following these steps (taken from https://docs.nersc.gov/connect/jupyter/ ). After creating the kernel, you just go to “Change Kernel” in the Jupyter notebook and you’re ready to go.

You can use one of our default Python 2, Python 3, or R kernels. If you have a Conda environment, depending on how it is installed, it may just show up in the list of kernels you can use. If not, use the following procedure to enable a custom kernel based on a Conda environment. Let's start by assuming you are a user with username user who wants to create a Conda environment on Cori and use it from Jupyter.

Be sure to specify what version of Python interpreter you want installed. This will create and install a JSON file called a "kernel spec" in kernel.json at the path described in the install command output.

Anvil/Blues

intel on anvil

First, you might want to build SCORPIO (see below), or use the one from @Xylar Asay-Davis referenced here:

SCORPIO on anvil

@Xylar Asay-Davis: If you need to compile it yourself, you can do that as follows:

Personal OSX Machine

Hyun-Gyu Kang: This personal approach worked for me (macOS Catalina 10.15).

• Required: Homebrew (https://brew.sh)

Installation of MPAS dependencies except PIO

PIO-1.9.23 installation with some modifications

MPAS-O installation

Personal Linux Machine

@Xylar Asay-Davis: This approach worked for me under Ubuntu 18.04

@Luke Van Roekel (Unlicensed) : Also worked for me under Max OS X 10.14.6

Installation of MPAS dependencies including SCORPIO and the compass conda environment

First, I run the following script in an empty directory that I can delete later:

Setup before compiling/running

Then, when I want to build or run MPAS-Ocean, I source a file containing:

Other Machines

This page replaces the page: https://acme-climate.atlassian.net/wiki/spaces/OCNICE/pages/130287880 , which has some old configurations for Titan and Theta. The old https://acme-climate.atlassian.net/wiki/spaces/OCNICE/pages/128814632 table of contents links to useful instructions for E3SM, analysis, and specific machines, but now may be out of date.

Library stack installation

Grizzly

Installation of PIO follows from the following pre-existing module files:

Note, DO NOT use openmpi/3.1.5 as there is a bug (RMIO https://github.com/MPAS-Dev/MPAS-Model/issues/576).

PIO2 from https://github.com/E3SM-Project/scorpio was used, specifically tag scorpio-v1.1.0 with the following build command (note use of intel compilers):

build with make and install with make install. Installation here implies

export PIO=/usr/projects/climate/SHARED_CLIMATE/software/grizzly/pio/1.10.1/intel-19.0.4/intel-mpi-2019.4/netcdf-4.7.3-parallel-netcdf-1.11.2/

as needed for the build.

References

• https://github.com/MPAS-Dev/MPAS-Model/issues/576

• https://github.com/E3SM-Project/scorpio/issues/308

MPAS within E3SM

E3SM has its own documentation, including tutorials and a quick start guide. Here we provide some abbreviated instructions relevant to MPAS components. The simplest way to get started with E3SM is

The MPAS source code is a submodule within E3SM, and is contained in the subdirectory components/mpas-source. The --recursive flag downloads MPAS code into that directory from the MPAS repository. If you cd into that directory, you can edit that code directly and use git commands on that repo locally.

Pre-packaged E3SM simulation: create_test

You can run a pre-packaged test case like this:

where ./create_test --help will show you all the options. Here PET is a thread test, Ln9 is for nine steps, T62 and oQU240 are the atmosphere and ocean resolution, and GMPAS-IAF is a G-case comp set. You can, for example, specify the machine, compiler, queue, walltime with

etc. if you don’t want the defaults. Note that query_config and query_testlists will list all the options for machines, resolutions, comp sets, etc.

Custom E3SM simulation: create_newcase

For more control over your simulation, use these four commands in sequence: create_newcase, case.setup, case.build, case.submit, where the later three reside in the case directory. See the E3SM documentation for more details. The create_test command in the previous section rolls these four together.

Here is an example of using these four commands, with bash variables to simplify the arguments.

where $E3SM_REPO is the path to your repository, $E3SM_CASE is the name of this particular simulation, and $CASE_ROOT is the path to your case directories on this particular machine.

To customize the processor layout, use the xmlchange command in the case directory, or edit the env_mach_pes.xml directly. For example,

The remaining steps are

At any time, you can change options for the next submission in env_run.xml. For example, to alter the duration, and restart write frequency

After the first run, you can set the next submission to continue from the restart, and auto-resubmit, with

E3SM Pull Requests involving MPAS code

The workflow for any contributor to alter MPAS code within E3SM is as follows:

1. If you are working on a certain CORE (ocean, seaice, landice) make a branch of MPAS-Dev/MPAS-Model:CORE/develop and make your code alterations.

2. Push that branch to your fork.

3. Make a pull request back to theCORE/develop branch of the MPAS-Dev/MPAS-Model repo.

A CORE developer is then responsible for the following steps:

1. Review the PR.

2. Merge into MPAS-Dev/MPAS-Model:CORE/develop

3. When the E3SM repo is available, merge CORE/develop into e3sm/develop.

4. In an E3SM repo, create a new branch from E3SM-Project/E3SM:master, and

5. Change any corresponding E3SM code. These files are often altered for defaults in
   namelists: components/mpas-ocean/bld/namelist_files/namelist_defaults_mpaso.xml
   streams file: components/mpas-ocean/cime_config/buildnml
   for the ocean core. See similar files for other cores.

6. Commit remaining changes.

7. Run a series of automated tests. A typical sequence on cori might include:

   This does not cover all possible combinations, but provides bit-for-bit threading (PET) and decomposition (PEM) tests on a variety of compsets, resolutions, compilers (intel/gnu) and machines (knl/haswell). The last two are a smoke test with debug on (SMS_D), and LISIO runs the MALI core, while the others do not. See instructions on this page, two sections up. To see the results, copy the path from the output line
   Creating test directory /PATH/TO/TEST/DIR
   and look for PASS versus FAIL with
   cat /PATH/TO/TEST/DIR/TestStatus

8. Push branch to E3SM-Project/E3SM repo. Branch names typically are formatted as username/CORE/topic, i.e. mark-petersen/ocean/best_change_ever.

9. Make a pull request from your new branch to E3SM-Project/E3SM:master.

10. An E3SM hub will review and merge. This is Jon Wolfe for MPAS changes. First the PR is merged to E3SM-Project/E3SM:next and tested with the E3SM nightly regression suite. If it passes, the PR is merged into E3SM-Project/E3SM:master.