E3SM Build System and CMake

Owned by James Foucar
Last updated: Nov 15, 2023
11 min read

For E3SM "core" components

A "core" component in this guide is a science model fully owned and developed by E3SM. At this time, these are:

  • cam
  • cice
  • clm
  • mosart

At a high-level, the diagram below describes how CIME handles these core components:

To make sense of the above diagrams, use this legend:

  • Squares = callstacks
  • Ovals = input/output files
  • Green = python
  • Blue = perl
  • Black = raw text
  • Purple = CMake
  • Pink = XML

Explaining a bit further, the build of an E3SM case is primarily influenced by the Setup (case.setup, represented by the left side of the diagram) and Build (case.build, represented by the right side of the diagram) case phases. Case.setup will copy cmake_macros from your repository to your case. These macros include a generic Macros.cmake file which uses case settings to include the various compiler and machine macros needed for the case. Macros.cmake will be copied directly into your case and the rest of the contents of cmake_macros will be recursively copied into $case/cmake_macros. These macro files contain things like which compilers to use, which compiler flags to use, and which link flags to add. The next key task for setup is building namelists for each active component. As best as I can tell, our builds are not influenced by namelist files, but the buildnml scripts are responsible for calling the build configure scripts ($component/bld/configure) which are still in Perl for all our core components. The build configure scripts are responsible for producing two key input files for the build phase, "Filepath" and "CCSM_cppdefs", which will be placed in $case/Buildconf/${component}conf (e.g. $case/Buildconf/camconf/Filepath). Filepath is list of directories that will be globbed for source files. An interesting feature of our build system is that no repeated file basenames are allowed. For example, if Filepath contains directories A and B and A/foo.f90 and B/foo.f90 both exist, then only A/foo.f90 will be compiled by our build system (assuming A came first in the Filepath list). I believe the original intent of this approach was to allow for custom per-case srcmods. CCSM_cppdefs is a list of c preprocessor directives that will be applied to every compilation of every source file in the component. Both Filepath and CCSM_cppdefs are tweakable by users (users can modify these files in-place and re-run case.build).

Getting to the build phase, case.build is now a thin wrapper around cmake with E3SM/components/CMakeLists.txt serving as the root cmake file. Details on how to use this file directly are below in the How-to guide.

Key Files

List of files that influence the build, both in our repo and in the case directory.

  • E3SM/cime_config/machines/config_machines.xml: Contains list of valid compilers (COMPILERS) and environment setup (modules, environment_variables).
  • E3SM/cime_config/machines/config_compilers.xml: DEPRECATED. Contains specifications (flags, link flags, exe names) for compilers (gcc, pgi) as well as machine-specific modifications for those compilers. This system can be reactivated by setting CIME_NO_CMAKE_MACRO=1 in your environment.
  • E3SM/cime_config/machines/cmake_macros/*: Contains specifications (flags, link flags, exe names) for compilers (gcc, pgi) as well as machine-specific modifications for those compilers. Replaces config_compilers.xml.
  • E3SM/cime_config/machines/Depends.$compiler.$machine.cmake: Contains machine/compiler-specific flag changes for specific E3SM component files
  • E3SM/cime_config/customize/e3sm_compile_wrap.py: The default compile/link wrapper. It is used to produce timings of all operations
  • E3SM/cime_config/machines/scripts/*: Python scripts to assist in managing cmake macro files.
  • E3SM/components/CMakeLists.txt: The root CMakeLists.txt file for E3SM, mostly just iterates over components and calls functions
  • E3SM/components/cmake/cmake_util.cmake: Contains some generally useful functions for cmake
  • E3SM/components/cmake/build_mpas_model.cmake: Implements the handoff between E3SM's CMake system to MPAS's Cmake system

  • E3SM/components/cmake/common_setup.cmake: Cmake setup common to all core components, this is only invoked ones (AFTER mpas is configured)

  • E3SM/components/cmake/build_model.cmake: Implements how an individual core component gets configures (involves reading the Filepath and CCSM_cppdefs, globbing files, etc)
  • E3SM/components/cmake/modules/*.cmake: Cmake modules for finding packages we depend on
  • E3SM/components/cmake/*/CMakeLists.txt: Very thin CMakeLists.txt files for each component type, this is just to ensure each component get it's own build area
  • E3SM/components/*/bld/configure: Perl configure script for component, should produce Filepath and CCSM_cppdefs files
  • E3SM/components/*/cime_config/buildnml: The buildnml script for component, controls what flags get passed to $component/bld/configure
  • E3SM/components/*/cime_config/buildlib: The legacy buildlib driver for the classic deprecated Makefile-based build system
  • E3SM/components/*/cime_config/buildlib_cmake: A insertion point for last-minute, component-specific changes before cmake is called. For example, mosart uses this file, not bld/configure, to generate Filepath and CCSM_cppdefs
  • E3SM/components/mpas-source/src/CMakeLists.txt: The root CMakeLists.txt for MPAS
  • E3SM/components/mpas-source/src/cmake_utils.cmake: Contains some generally useful functions for configuring MPAS
  • E3SM/components/mpas-source/src/build_core.cmake: A generic implementation for building MPAS cores
  • E3SM/components/mpas-source/src/core_landice/landice.cmake: The CMake definition of the landice core
  • E3SM/components/mpas-source/src/core_ocean/ocean.cmake: The CMake definition of the ocean core
  • E3SM/components/mpas-source/src/core_seaice/seaice.cmake: The CMake definition of the seaice core
  • E3SM/share/buildlib: Python build wrapper scripts for our SHAREDLIB packages. CIME invokes these during the SHAREDLIB build phase.
  • $case/Buildconf/*conf: The various build conf dirs for all active components, the Filepath and CCSM_cppdefs files for each component live here
  • $case/Macros.cmake: The file that includes the relevant files in cmake_macros. You can modify this locally in your case.
  • $case/cmake_macros/*: The files that ultimately sets flags, link flags, etc. You can modify this locally in your case.
  • $case/Macros.make: A makefile macro derived from the cmake macros, needed for the sharedlibs that still use Makefile build systems. This is an auto-generated file that you should never edit.
  • $case/SourceMods/src.$component/*: Alternative source files to use instead of the ones from the component in your source dir
  • $EXEROOT/cmake-bld: The root cmake build/binary directory 
  • $case/Tools/e3sm_compile_wrap.py: The case-specific compiler wrapper, this will start out as a copy of the e3sm_compile_wrap.py from the repo. Users can customize this one.

MPAS

Homme is built as a part of cam, so that leaves our MPAS components as the remaining components that need special consideration in our build system. MPAS was always built very differently than our internal components and I had to add a CMake build system to MPAS to reflect that. TODO: explain more, or separate page?

Development cycle

Build, fix, build, fix cycle. Best way. Testing tweaking of compiler flags and link flags.

How-to guide for common actions

Invoking CMake directly

There could be many reasons to do this. You may want to do many rounds of configuring without having to go through CIME (case.build) every time if, for example, you were debugging something cmake-specific). You may also want to play around with tweaking the arguments being passed to CMake.

The cmake command will be printed near the top of the e3sm build log (located in $EXEROOT/e3sm.bldlog.*). If you copy this command (you don't need (and probably don't want) to copy the output redirection stuff at the end), you can 'cd $EXEROOT/cmake-bld; paste cmake cmd' and you'll then be configuring E3SM just like case.build would. Be sure not to forget to source $case/.env_mach_specific.sh first so that your environment matches the CIME build environment.

Note: the e3sm.bldlog will only exist if you have previously attempted a build in your case. If you want to go straight to doing a direct build:

% ./create_test $test_case --no-build # Could also do create_newcase here instead
% cd $case_dir
% ./case.setup                        # only needed if you did create_newcase instead of create_test
% . .env_mach_specific.sh             # set your shell env to match E3SM machine/compiler env
% ./case.build --sharedlib-only       # make sure sharedlibs are built, these are not yet handled by cmake
% ./case.build --model-only --dry-run
...
* look for CMake cmd, you can also find this command near the top of the e3sm.bldlog for cases that have already run through the build phase*
...
% cd $exeroot/cmake-bld
% * copy/paste cmake cmd and run *


Invoking make (or ninja) directly

If doing development and fixing compile errors, it would probably be most efficient to run make directly rather than going through case.build. Be sure not to forget to source $case/.env_mach_specific.sh first so that your environment matches the CIME build environment. 'cd $EXEROOT/cmake-bld' and just run 'make' ( you can add -jN of course too). Note: once the build is complete, you'll still need to run case.build in order for CIME bookkeeping to occur; it should only take a few seconds.

Examining flag settings for a component

Assuming you are in $EXEROOT/cmake-bld, a nice synopsis of the flag settings for a component can be found in cmake/$comp_class/CMakeFiles/$comp_class.dir/flags.make

For example, for atm:

% cat cmake/atm/CMakeFiles/atm.dir/flags.make

This will nicely show default flags for each language, includes, cpp defs, and per-file flag customizations.

Examining link flags for E3SM

Leaving aside the static lib creation, there's only one interesting link command in our build: the linking of the e3sm.exe executable. You can see what this command will be by looking at cmake/cpl/CMakeFiles/e3sm.exe.dir/link.txt

Temporarily tweaking compiler flags for a broken file

Be sure not to forget to source $case/.env_mach_specific.sh first so that your environment matches the CIME build environment. 'cd $EXEROOT/cmake-bld && make'. Copy the build line for the broken compilation from the build log file and paste it into your terminal and make tweaks.

Be sure not to forget to source $case/.env_mach_specific.sh first so that your environment matches the CIME build environment. 'cd $EXEROOT/cmake-bld' . Just find and edit the appropriate link.txt file (will typically be  cmake/cpl/CMakeFiles/e3sm.exe.dir/link.txt, edit, and re-run make).

Temporarily tweaking compiler flags for an entire component

For example, for eam:

% edit $case/cmake_macros/$compiler_$mach.cmake
* Add cmake code to check component name and make the corresponding change, e.g.
if (COMP_NAME STREQUAL eam)
  string(APPEND CMAKE_Fortran_FLAGS " -vector0")
endif()
% make

Temporarily tweak CPP defs for a component

Just edit the $case/Buildconf/${component}conf/CIME_cppdefs file and re-run either case.build or the appropriate cmake command

Temporarily tweak Filepaths for a component

Just edit the $case/Buildconf/${component}conf/Filepath file and re-run either case.build or the appropriate cmake command

Temporarily tweak compilation or link flags for the entire build

Just edit the $case/cmake_macros/$compiler_$mach.cmake file and run case.build or make

Adding/removing files to be compiled

Since our system uses globs, just add or remove files from any directory in Filepath and re-run the build.

Permanently change CPP defs for a component

You'll need to edit either E3SM/components/$component/bld/configure OR E3SM/components/$component/cime_config/buildlib_cmake and make a PR to merge your change.

Permanently change compiler flags for a component or entire build (How to use our CMake macro system!)

CMake macros are loaded in this order. If the macro doesn't exist, that level is skipped. Users will usually be working with the $compiler_$mach.cmake macro.

  • universal.cmake : applies changes globally
  • $compiler.cmake : applies changes for a compiler type (on all machines)
  • $mach.cmake : applies changes for a machine (for all compilers)
  • $compiler_$mach.cmake : applies changes to a compiler/mach comb
  • post_process.cmake: finishes setting up cmake stuff, developers should not edit this file

You can write any cmake you want in these. The macros have access to all the key case settings which can be used for conditionals. The common variables to set/adjust in $compilers_$mach.cmake are similiar to what any CMake project would use, the only difference is that we don't set these in the CACHE (allows each component to customize). You can set any CMake variable in these macros that you want, for example:

  • Language compilation flags CMAKE_<LANG>_FLAGS[_<BUILD_TYPE>], examples:
    • CMAKE_C_FLAGS: flags for compiling C source files
    • CMAKE_C_FLAGS_DEBUG: flags for compiling C sources files for DEBUG builds
    • CMAKE_Fortran_FLAGS: flags for compiler fortran source files
    • CMAKE_CXX_FLAGS: flags for compiling CXX source files

  • CMAKE_EXE_LINKER_FLAGS: link flags 

  • There are tons of CMake variables available to tweak your builds: https://cmake.org/cmake/help/latest/manual/cmake-variables.7.html


  • There are a number of variables that are specific to E3SM that are also available:
    • CPPDEFS: c preprocessor definitions for all builds.
    • CPPDEFS_DEBUG: c preprocessor definitions added for DEBUG buids.
    • The serial/mpi compiler names (SCC, SCXX, SFC, MPICC, MPICXX, MPIFC). We still need these (instead of just using CMAKE_<LANG>_COMPILER ) because some of our build system needs to know about serial vs. MPI compilers

    • SPIO_CMAKE_OPTS: cmake options to pass along to your scorpio build

    • PIO_FILESYSTEM_HINTS: this setting also gets passed to scorpio

    • USE_HIP / USE_CUDA: enable gpu

    • E3SM_LINK_WITH_FORTRAN: tells cmake to use the fortran compiler to link the e3sm.exe. Some languages must use fortran to link a program that has a fortran main function

    • CONFIG_ARGS: autoconf configure arguments to pass to sharedlibs that use autoconf (MCT and mpi-serial)

    • KOKKOS_OPTIONS: cmake options to pass along to your kokkos build

For the first time, our build system supports conditionals based on versions (instead of having to make entirely new compiler definitions). Example:

if (CMAKE_Fortran_COMPILER_VERSION VERSION_GREATER_EQUAL 10)
   string(APPEND CMAKE_Fortran_FLAGS " -fallow-argument-mismatch  -fallow-invalid-boz ")
endif()

More details about the cmake macro system can be found here: https://github.com/E3SM-Project/E3SM/pull/4537

As an alternative to version conditionals, you can manually include any macro you want. As an example, $mach_$compiler$vers.cmake could include $mach_$compiler.cmake via:

include("${MACROS_DIR}/<mach>_<compiler>.cmake")

Permanently change Filepaths for a component

You'll need to edit either E3SM/components/$component/bld/configure OR E3SM/components/$component/cime_config/buildlib_cmake and make a PR to merge your change.

Permanently change macro settings based on the presence of a component

You can parse the entire compset (as opposed to just looking at the current component) by putting something like the following in the relevant cmake macro:

if ("name-of-component" IN_LIST COMP_NAMES)
  # Make config change here
endif()

This can be useful when a global change is needed based on the presence of a component. A potential use case is change the configuration of a shared lib based on the presence of a component. For example, to change the Kokkos configuration if eamxx is an active component:

if ("eamxx" IN_LIST COMP_NAMES)
  string(APPEND KOKKOS_OPTIONS " -DKokkos_ENABLE_OPENMP=ON")
endif()

Permanently add compile flags for a specific file

You'll need to edit E3SM/components/cmake/build_model.cmake (or the appropriate Depends file) as follows:

if (COMP_NAME STREQUAL "name-of-component")
  e3sm_add_flags("${SOURCE_FILE}" "-flag1 -flag2 ...")
endif()

You'll need to refer to the source file via it's relative path relative to E3SM/components, e.g. cam/src/physics/rrtmg/ext/rrtmg_lw/rrtmg_lw_k_g.f90

Permanently change compile flags for a generated file (.F90.in)

You'll need to edit E3SM/components/cmake/build_model.cmake (or the appropriate Depends file) as follows:

if (COMP_NAME STREQUAL "name-of-component")
  e3sm_add_flags("${CMAKE_CURRENT_BINARY_DIR}/${BASEFILENAME}.F90" "-flag1 -flag2 ...")
endif()

Permanently disable optimizations for a specific file

This is a fairly common occurrence. We have a number of files in E3SM that would take a very long time to compile with optimizations on. You'll need to edit E3SM/components/cmake/build_model.cmake (if you want to deoptimize a file universally (on all platforms)) or in the Depends files (if you want to deoptimize for specific platforms) as follows:

if (COMP_NAME STREQUAL "name-of-component")
  list(APPEND NOOPT_FILES "file1" "file2" ...)
endif()

Permanently change compile flags for a specific file for only specific compilers/machines for E3SM component source files

The three actions above can also be made to E3SM/cime/config/e3sm/machines/Depends.$compiler.$machine.cmake in order to only impact that compiler/machine. Note, there can be some confusion here because the Depends.* filename can have a compiler name, a machine name, or both. All matching Depends.* files will be loaded for a build. The order in which these Depends files are processed is: Depends.$compiler, Depends.$machine, Depends.$compiler.$machine. 

Permanent change compiler flags for sharedlib source files

Our sharedlib builds still use the class Makefile-based system to drive the builds for most of our sharedlibs. If machine/compiler-specific flag tweaks are needed, you can use the same Depends.* system described immediately above, except be sure to omit the .cmake suffix for the filename and use Makefile syntax.

Clean my build

'cd $EXEROOT/cmake-bld; make clean'

Totally clean my build

'cd $EXEROOT/cmake-bld; rm -rf *'

Build only a specific component

'cd $EXEROOT/cmake-bld; make $comp_class' e.g ("atm")

Clean only a specific component

'cd $EXEROOT/cmake-bld/cmake/$comp_class; make clean'

Customizing build wrapper actions and timings

As of recent (2020) PRs, the E3SM build system will always wrap compilations and links with the e3sm_compile_wrap.py script. The default version of this script lives in E3SM/cime_config/customize/e3sm_compile_wrap.py and will be copied to all cases in $case/Tools. The one in the case is what actually gets used by the build system and is modifiable by the user at any time. The default behavior of this script is to time add timing info for all links and compilations to the e3sm.bldlog. A handy command for listing actions from cheapest to most expensive: `grep "built in" e3sm.bldlog.* | sort -n -k5,5` . This makes it very easy to find expensive actions.

Using SourceMods

The SourceMods mechanism is a way of bundling code changes within a case. We only recommend using SourceMods for small, short-term changes that just apply to one or two cases. For larger or longer-term changes, including gradual, incremental changes towards a final solution, we highly recommend making changes in the main source tree, leveraging version control (git). The SourceMods directory lives under the case and has subdirectories for each active component, example $case/SourceMods/src.eam. For any source file $name.$suffix in the source mod dir for a component, that file will take "precedence" over files of the same name in repo for that component. Our build system will automatically detect SourceMods changes and will know it needs to reconfigure (recreate cache files) when changes to SourceMods are made.

CSM_Share

This code lives in E3SM/share and is built during the sharedlibs build. It has a cmake build system very similar to one of our core components. Depends files for csm_share live in E3SM/share and need to be kept separate from the "main" Depends files.

Using custom TPLs/sharedlibs

Developers can force the build system to use custom installations of E3SM dependencies by making changes to ${PACKAGE_NAME}_ROOT in their shell environment. As an example, if you set Trilinos_ROOT, your build will build against that Trilinos.