Developing Documentation for E3SM

Owned by Robert Jacob
Last updated: about 4 hours ago
5 min read

The E3SM source code includes the documentation of the model in text files stored along with source code in the main repository.

Like any other new development in the repository, documentation additions should be made on a feature branch and added to the repo with a pull request as described in .

Our documentation is written in Markdown language. See https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax to get started. NOTE: the Markdown files are converted to html. They will look different if you view them directly with a Markdown renderer.

Doing local development of documentation

Just like you compile and test your Fortran before committing it, you should “compile” and view your documentation before committing and pushing the branch. Unlike compiling and running the model, the documentation can be easily built and displayed on your laptop. But first you need to install the python programs needed to build the documentation.

Set up documentation development environment.

You only have to do this once on each machine you wish to use for writing documentation.

Python virtual environment

On your local laptop in your home directory, create a python virtual environment called “mkdocsenv”

python3 -m venv mkdocsenv (you can call it something else besides “mkdocsenv”)

Now enter that environment with

source mkdocsenv/bin/activate

Install the python packages needed by entering these commands:

pip install mkdocs-material mkdocs-monorepo-plugin pymdown-extensions mdutils mkdocs-bibtex mkdocs-redirects

Leave the environment by typing

deactivate

The creation of mkdocsenv and the pip install commands only have to be done once on each platform you want to do documentation development.

Installing a python package in a virtual environment keeps it from being installed system wide which is good practice and also useful if you don’t have permissions to install system-level packages.

Conda

The same python environment above can be achieved with conda.

To do so, create a conda environment with

conda create -n e3sm_docs mkdocs-material pymdown-extensions mkdocs-monorepo-plugin mdutils mkdocs-bibtex mkdocs-redirects markdownlint-cli2

activate and use the environment with:

conda activate e3sm_docs to use the environment.

Use local environment to build/test documentation changes

Activate your environment that you made in the above step.

source mkdocsenv/bin/activate OR conda activate e3sm_docs

cd to the top level E3SM directory and see if you can build the current docs by typing

mkdocs build

EAMxx docs use an automated routine to populate the namelist parameters in its documentation. In order to avoid warnings during build of the complete documentation, please ensure you have a copy of the CIME submodule inside your main repo. Proceed to executing eamxx-params-docs-autogen in the components/eamxx/scripts directory. Then, you will be able to build with mkdocs build --strict --verbose (to convert any warning into an error, which we use in the PR checking).

View the docs locally in your browser with the built-in server

mkdocs serve

(Follow the instructions output by the above command to display in your browser. control-C to end server)

NOTE: Don’t rely on direct Markdown viewers to see how things look. You need to look at the derived html because that is what will be served at github.io.

An alternative to mkdocs serve is to access to the docs directly without serving them through a URL. To achieve this, one could you file:///E3SM_ROOT/site/index.html in a modern browser like Chrome (in place of a URL). In this case, E3SM_ROOT is the full path to the clone E3SM repository on your local machine where you built the docs. One could directly go to other pages as needed using the same method (e.g., file:///E3SM_ROOT/site/EAM/index.html). Note that as you click on links that are in subdirectories, you’ll see the list of files instead of the rendered content. Click on “index.html” when that happens.

As you edit the documentation, rerun the “mkdocs build” command and refresh the website in your browser. The output of “mkdocs build” will tell you if you have a syntax error in your markdown files. The website in your browser will let you know if it looks as you intended.

Using e3sm-unified to build/test docs

If you prefer to avoid using your local machine to build and test the docs, you can use one of the supported machines to do so and you do not need to install anything described in the previous section. Example annotated instructions to use Chrysalis:

  • tunnel a port to chrysalis from your local machine (e.g., ssh -L 12345:localhost:12345 username@chrysalis.lcrc.anl.gov)

  • use e3sm-unified (e.g., source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh)

  • develop the docs in your preferred editor

  • (optionally, starting in 1.10.0, use the linter, e.g., markdownlint-cli2 <my_md_files>)

  • starting in 1.9.3, use mkdocs serve with that port (e.g., mkdocs serve -a localhost:12345)

  • then, open the link it prints in your local browser (e.g., localhost:12345)

  • (note that you can continue editing and your edits should appear live in the browser window very shortly after making the edits)

Markdown style and linting

To help with maintainability, everyone should follow the rules in markdownlint for their markdown files. The rules used are available at https://github.com/DavidAnson/markdownlint?tab=readme-ov-file#rules--aliases

To enforce those rules, any PR that adds are modifies markdown files will check them with a linter. (A linter is a tool for processing source code and looking for problems or enforcing a style and is named after the “lint” tool for C.) E3SM uses the widely used “markdownlint-cli2” linter through a github action. The PR will be blocked until the all the rules are followed (except for line length).

You can install the linter on your local machine by adding it to the conda environment above or following the instructions at https://github.com/DavidAnson/markdownlint-cli2 (using homebrew, npm or a docker container). To run it, use markdownlint-cli2 <file-or-glob> where <file-or-glob> is markdown file (e.g., my_docs.md) or a glob of files (e.g., my_new_docs/**/*.md).

Documentation organization

The documentation for a model should be stored within that model’s source code directory.

The first file needed is mkdocs.yml which should be in E3SM/components/<model>. See E3SM/components/eam/mkdocs.yml for an example:

site_name: EAM nav: - Introduction: 'index.md' - Users's Guide: user-guide/index.md - Developers's Guide: dev-guide/index.md - Technical Guide: tech-guide/index.md

The “site_name” is a keyword used in the E3SM/docs/index.md file to reference all the model’s documentation. (That file must be edited when adding a new model’s documentation).

All other files should be stored under E3SM/components/<model>/docs

The basic structure under “docs” is

index.md - top level introductory text and links to main sections dev-guide/ - subdir for development guide tech-guide/ - subidr for technical guide user-guide/ - subdir for user guide

Each of the sub-dirs should also have an “index.md” file to organize their content.

Files and sections

A subsection of your documentation should be contained entirely in a single file. Short sections can be grouped in to one file.

Equation numbering and references to equations are local to an individual markdown file.

Figures

When you start adding figures to your documentation, keep in mind that binary files can lead to repo size bloat IF they change a lot so don’t commit a figure to the repo until you are confident its the near-final version.

Figures should be in png format and under 500 pixels on the longest side.

Keep figures in the same directory as the Markdown file referencing them.

Automated preview during PRs

When you submit a PR, we have an automated process to build and test the docs (with --strict --verbose) and to preview them directly in the PR. An automated bot will leave a comment with a URL for you to review the docs.