...
At a high-level, the diagram below describes how CIME handles these core components:
To make sense of the above diagramdiagrams, use this legend:
- Squares = callstacks
- Ovals = input/output files
- Green = python
- Blue = perl
- Black = raw text
- Purple = CMake
- Arrow out = Produces
- Arrow in = Consumes/uses
Explaining a bit further, the build of an E3SM case is primarily influenced by the Setup (case.setup, represented by the left side of the diagram) and Build (case.build, represented by the right side of the diagram) case phases. Case.setup will call BuildTools.configure, which processes all the XML stuff in E3SM/cime/config/e3sm/machines/config_compilers.xml and produces a Macros.cmake file. This file contains things like which compilers to copy cmake_macros from your repository to your case. These macros include a generic Macros.cmake file which uses case settings to include the various compiler and machine macros needed for the case. Macros.cmake will be copied directly into your case and the rest of the contents of cmake_macros will be recursively copied into $case/cmake_macros. These macro files contain things like which compilers to use, which compiler flags to use, and which link flags to add. The next key task for setup is building namelists for each active component. As best as I can tell, our builds are not influenced by namelist files, but the buildnml scripts are responsible for calling the build configure scripts ($component/bld/configure) which are still in Perl for all our core components. The build configure scripts are responsible for producing two key input files for the build phase, "Filepath" and "CCSM_cppdefs", which will be placed in $case/Buildconf/${component}conf (e.g. $case/Buildconf/camconf/Filepath). Filepath is list of directories that will be globbed for source files. An interesting feature of our build system is that no repeated file basenames are allowed. For example, if Filepath contains directories A and B and A/foo.f90 and B/foo.f90 both exist, then only A/foo.f90 will be compiled by our build system (assuming A came first in the Filepath list). I believe the original intent of this approach was to allow for custom per-case srcmods. CCSM_cppdefs is a list of c preprocessor directives that will be applied to every compilation of every source file in the component. Both Filepath and CCSM_cppdefs are tweakable by users (users can modify these files in-place and re-run case.build).
...
- E3SM/cime_config/machines/config_machines.xml: Contains list of valid compilers (COMPILERS) and environment setup (modules, environment_variables).
- E3SM/cime_config/machines/config_compilers.xml: DEPRECATED. Contains specifications (flags, link flags, exe names) for compilers (gcc, pgi) as well as machine-specific modifications for those compilers. E3SM/cime_config/machines/Depends.$compiler.$machine.cmake: Contains machine/compiler-specific flag changes for specific E3SM component filesThis system can be reactivated by setting CIME_NO_CMAKE_MACRO=1 in your environment.
- E3SM/cime_config/machines/Depends.$compiler.$machine: Contains machine/compiler-specific flag changes for shared-lib files, this system still uses Makefile syntax/cmake_macros/*: Contains specifications (flags, link flags, exe names) for compilers (gcc, pgi) as well as machine-specific modifications for those compilers. Replaces config_compilers.xml.
- E3SM/cime_config/scripts/Tools/e3sm_compile_wrap.py: The default compile/link wrapper. machines/Depends.$compiler.$machine.cmake: Contains machine/compiler-specific flag changes for specific E3SM component files
- E3SM/cime_config/customize/e3sm_compile_wrap.py: The default compile/link wrapper. It is used to produce timings of all operations
- E3SM/cime_config/machines/scripts/*: Python scripts to assist in managing cmake macro files.
- E3SM/components/CMakeLists.txt: The root CMakeLists.txt file for E3SM, mostly just iterates over components and calls functions
- E3SM/components/cmake/cmake_util.cmake: Contains some generally useful functions for cmake
- E3SM/components/cmake/build_mpas_model.cmake: Implements the handoff between E3SM's CMake system to MPAS's Cmake system
E3SM/components/cmake/common_setup.cmake: Cmake setup common to all core components, this is only invoked ones (AFTER mpas is configured)
- E3SM/components/cmake/build_model.cmake: Implements how an individual core component gets configures (involves reading the Filepath and CCSM_cppdefs, globbing files, etc)
- E3SM/components/cmake/modules/*/CMakeLists.txt: Very thin CMakeLists.txt cmake: Cmake modules for finding packages we depend on
- E3SM/components/cmake/*/CMakeLists.txt: Very thin CMakeLists.txt files for each component type, this is just to ensure each component get it's own build area
- E3SM/components/*/bld/configure: Perl configure script for component, should produce Filepath and CCSM_cppdefs files
- E3SM/components/*/cime_config/buildnml: The buildnml script for component, controls what flags get passed to $component/bld/configure
- E3SM/components/*/cime_config/buildlib: The legacy buildlib driver for the classic deprecated Makefile-based build system
- E3SM/components/*/cime_config/buildlib_cmake: A insertion point for last-minute, component-specific changes before cmake is called. For example, mosart uses this file, not bld/configure, to generate Filepath and CCSM_cppdefs
- E3SM/components/mpas-source/src/CMakeLists.txt: The root CMakeLists.txt for MPAS
- E3SM/components/mpas-source/src/cmake_utils.cmake: Contains some generally useful functions for configuring MPAS
- E3SM/components/mpas-source/src/build_core.cmake: A generic implementation for building MPAS cores
- E3SM/components/mpas-source/src/core_landice/landice.cmake: The CMake definition of the landice core
- E3SM/components/mpas-source/src/core_ocean/ocean.cmake: The CMake definition of the ocean core
- E3SM/components/mpas-source/src/core_seaice/seaice.cmake: The CMake definition of the seaice core
- E3SM/share/buildlib: Python build wrapper scripts for our SHAREDLIB packages. CIME invokes these during the SHAREDLIB build phase.
- $case/Buildconf/*conf: The various build conf dirs for all active components, the Filepath and CCSM_cppdefs files for each component live here
- $case/Macros.cmake: The file that ultimately sets flags, link flags, etcincludes the relevant files in cmake_macros. You can modify this locally in your case. $EXEROOT
- $case/cmake-bld: The root cmake build/binary directory
- $case/Tools/e3sm_compile_wrap.py: The case-specific compiler wrapper, this will start out as a copy of the e3sm_compile_wrap.py from the repo. Users can customize this one.
MPAS
- _macros/*: The files that ultimately sets flags, link flags, etc. You can modify this locally in your case.
- $case/Macros.make: A makefile macro derived from the cmake macros, needed for the sharedlibs that still use Makefile build systems. This is an auto-generated file that you should never edit.
- $case/SourceMods/src.$component/*: Alternative source files to use instead of the ones from the component in your source dir
- $EXEROOT/cmake-bld: The root cmake build/binary directory
- $case/Tools/e3sm_compile_wrap.py: The case-specific compiler wrapper, this will start out as a copy of the e3sm_compile_wrap.py from the repo. Users can customize this one.
MPAS
Homme is built as a part of cam, so that leaves our MPAS components as the remaining components that need special consideration in our build system. MPAS was always built very differently than our internal components and I had to add a CMake build system to MPAS to reflect that. TODO: explain more, or separate page?
...
| Code Block |
|---|
% ./create_test $test_case --no-build # Could also do create_newcase here instead % cd $case_dir % ./case.setup # only needed if you did create_newcase instead of create_test % . .env_mach_specific.sh # set your shell env to match E3SM machine/compiler env % ./case.build --sharedlib-only # make sure sharedlibs are built, these are not yet handled by cmake % ./case.build --model-only --dry-run ... * look for CMake cmd*, ... you can also find this command near the top of the e3sm.bldlog for cases that have already run through the build phase* ... % cd $exeroot/cmake-bld % * copy/paste cmake cmd and run * |
...
Be sure not to forget to source $case/.env_mach_specific.sh first so that your environment matches the CIME build environment. 'cd $EXEROOT/cmake-bld && make'. Copy the build line for the broken compilation from the build log file and paste it into your terminal and make tweaks.
...
Temporarily tweaking compiler flags for an entire component
For example, for atmeam:
| Code Block |
|---|
% edit $case/cmake/atm/CMakeFiles/atm.dir/flags.make % make |
Temporarily tweak CPP defs for a component
Just edit the $case/Buildconf/${component}conf/CIME_cppdefs file and re-run either case.build or the appropriate cmake command
...
_macros/$compiler_$mach.cmake
* Add cmake code to check component name and make the corresponding change, e.g.
if (COMP_NAME STREQUAL eam)
string(APPEND CMAKE_Fortran_FLAGS " -vector0")
endif()
% make |
Temporarily tweak CPP defs for a component
Just edit the $case/Buildconf/${component}conf/Filepath CIME_cppdefs file and re-run either case.build or the appropriate cmake command
Temporarily tweak
...
Filepaths for a component
Just edit the $case/Macros.cmake Buildconf/${component}conf/Filepath file and re-run either case.build . You will need to run case.build --clean if you've already or the appropriate cmake command
Temporarily tweak compilation or link flags for the entire build
Just edit the $case/cmake_macros/$compiler_$mach.cmake file and run case.build before the change.or make
Adding/removing files to be compiled
...
You'll need to edit either E3SM/components/$component/bld/configure OR E3SM/components/$component/cime_config/buildlib_cmake and make a PR to merge your change.
Permanently change compiler flags for a component
...
You can use the COMP_NAME and COMP_CLASS selectors in our config_compilers.xml:
| Code Block |
|---|
<FFLAGS>
<append COMP_NAME="cam"> -qspill=6000 </append>
</FFLAGS> |
Permanently change Filepaths for a component
You'll need to edit either E3SM/components/$component/bld/configure OR E3SM/components/$component/cime_config/buildlib_cmake and make a PR to merge your change.
Permanently add compile flags for a specific file
You'll need to edit E3SM/components/cmake/build_model.cmake (or the appropriate Depends file) as follows:
| Code Block |
|---|
if (COMP_NAME STREQUAL "name-of-component")
e3sm_add_flags("${SOURCE_FILE}" "-flag1 -flag2 ...")
endif() |
You'll need to refer to the source file via it's relative path relative to E3SM/components, e.g. cam/src/physics/rrtmg/ext/rrtmg_lw/rrtmg_lw_k_g.f90
Permanently remove compile flags for a specific file
You'll need to edit E3SM/components/cmake/build_model.cmake (or the appropriate Depends file) as follows:
| Code Block |
|---|
if (COMP_NAME STREQUAL "name-of-component")
e3sm_remove_flags("${SOURCE_FILE}" "-flag1 -flag2 ...")
endif() |
Permanently change compile flags for a generated file (.F90.in)
You'll need to edit E3SM/components/cmake/build_model.cmake (or the appropriate Depends file) as follows:
| Code Block |
|---|
if (COMP_NAME STREQUAL "name-of-component")
e3sm_add_flags("${CMAKE_CURRENT_BINARY_DIR}/${BASEFILENAME}.F90" "-flag1 -flag2 ...")
endif() |
Permanently disable optimizations for a specific file
This is a fairly common occurrence. We have a number of files in E3SM that would take a very long time to compile with optimizations on. You'll need to edit E3SM/components/cmake/build_model.cmake as follows:
| Code Block |
|---|
if (COMP_NAME STREQUAL "name-of-component")
list(APPEND NOOPT_FILES "file1" "file2" ...)
endif() |
Permanently change compile flags for a specific file for only specific compilers/machines for E3SM component source files
The three actions above can also be made to E3SM/cime/config/e3sm/machines/Depends.$compiler.$machine.cmake in order to only impact that compiler/machine. Note, there can be some confusion here because the Depends.* filename can have a compiler name, a machine name, or both. All matching Depends.* files will be loaded for a build. The order in which these Depends files are processed is: Depends.$compiler, Depends.$machine, Depends.$compiler.$machine.
Permanent change compiler flags for sharedlib source files
Our sharedlib builds still use the class Makefile-based system to drive the builds for most of our sharedlibs. If machine/compiler-specific flag tweaks are needed, you can use the same Depends.* system described immediately above, except be sure to omit the .cmake suffix for the filename and use Makefile syntax.
Clean my build
'cd $EXEROOT/cmake-bld; make clean'
Totally clean my build
'cd $EXEROOT/cmake-bld; rm -rf *'
Build only a specific component
'cd $EXEROOT/cmake-bld; make $comp_class' e.g ("atm")
Clean only a specific component
'cd $EXEROOT/cmake-bld/cmake/$comp_class; make clean'
Customizing build wrapper actions and timings
...
or entire build (How to use our CMake macro system!)
CMake macros are loaded in this order. If the macro doesn't exist, that level is skipped. Users will usually be working with the $compiler_$mach.cmake macro.
- universal.cmake : applies changes globally
- $compiler.cmake : applies changes for a compiler type (on all machines)
- $mach.cmake : applies changes for a machine (for all compilers)
- $compiler_$mach.cmake : applies changes to a compiler/mach comb
- post_process.cmake: finishes setting up cmake stuff, developers should not edit this file
You can write any cmake you want in these. The macros have access to all the key case settings which can be used for conditionals. The common variables to set/adjust in $compilers_$mach.cmake are similiar to what any CMake project would use, the only difference is that we don't set these in the CACHE (allows each component to customize). You can set any CMake variable in these macros that you want, for example:
- Language compilation flags CMAKE_<LANG>_FLAGS[_<BUILD_TYPE>], examples:
- CMAKE_C_FLAGS: flags for compiling C source files
- CMAKE_C_FLAGS_DEBUG: flags for compiling C sources files for DEBUG builds
- CMAKE_Fortran_FLAGS: flags for compiler fortran source files
- CMAKE_CXX_FLAGS: flags for compiling CXX source files
CMAKE_EXE_LINKER_FLAGS: link flags
- There are tons of CMake variables available to tweak your builds: https://cmake.org/cmake/help/latest/manual/cmake-variables.7.html
- There are a number of variables that are specific to E3SM that are also available:
- CPPDEFS: c preprocessor definitions for all builds.
- CPPDEFS_DEBUG: c preprocessor definitions added for DEBUG buids.
- The serial/mpi compiler names (SCC, SCXX, SFC, MPICC, MPICXX, MPIFC). We still need these (instead of just using CMAKE_<LANG>_COMPILER ) because some of our build system needs to know about serial vs. MPI compilers
SPIO_CMAKE_OPTS: cmake options to pass along to your scorpio build
PIO_FILESYSTEM_HINTS: this setting also gets passed to scorpio
- USE_HIP / USE_CUDA: enable gpu
E3SM_LINK_WITH_FORTRAN: tells cmake to use the fortran compiler to link the e3sm.exe. Some languages must use fortran to link a program that has a fortran main function
- CONFIG_ARGS: autoconf configure arguments to pass to sharedlibs that use autoconf (MCT and mpi-serial)
KOKKOS_OPTIONS: cmake options to pass along to your kokkos build
For the first time, our build system supports conditionals based on versions (instead of having to make entirely new compiler definitions). Example:
| Code Block |
|---|
if (CMAKE_Fortran_COMPILER_VERSION VERSION_GREATER_EQUAL 10)
string(APPEND CMAKE_Fortran_FLAGS " -fallow-argument-mismatch -fallow-invalid-boz ")
endif() |
More details about the cmake macro system can be found here: https://github.com/E3SM-Project/E3SM/pull/4537
As an alternative to version conditionals, you can manually include any macro you want. As an example, $mach_$compiler$vers.cmake could include $mach_$compiler.cmake via:
| Code Block |
|---|
include("${MACROS_DIR}/<mach>_<compiler>.cmake") |
Permanently change Filepaths for a component
You'll need to edit either E3SM/components/$component/bld/configure OR E3SM/components/$component/cime_config/buildlib_cmake and make a PR to merge your change.
Permanently change macro settings based on the presence of a component
You can parse the entire compset (as opposed to just looking at the current component) by putting something like the following in the relevant cmake macro:
| Code Block |
|---|
if ("name-of-component" IN_LIST COMP_NAMES)
# Make config change here
endif() |
This can be useful when a global change is needed based on the presence of a component. A potential use case is change the configuration of a shared lib based on the presence of a component. For example, to change the Kokkos configuration if eamxx is an active component:
| Code Block |
|---|
if ("eamxx" IN_LIST COMP_NAMES)
string(APPEND KOKKOS_OPTIONS " -DKokkos_ENABLE_OPENMP=ON")
endif() |
Permanently add compile flags for a specific file
You'll need to edit E3SM/components/cmake/build_model.cmake (or the appropriate Depends file) as follows:
| Code Block |
|---|
if (COMP_NAME STREQUAL "name-of-component")
e3sm_add_flags("${SOURCE_FILE}" "-flag1 -flag2 ...")
endif() |
You'll need to refer to the source file via it's relative path relative to E3SM/components, e.g. cam/src/physics/rrtmg/ext/rrtmg_lw/rrtmg_lw_k_g.f90
Permanently change compile flags for a generated file (.F90.in)
You'll need to edit E3SM/components/cmake/build_model.cmake (or the appropriate Depends file) as follows:
| Code Block |
|---|
if (COMP_NAME STREQUAL "name-of-component")
e3sm_add_flags("${CMAKE_CURRENT_BINARY_DIR}/${BASEFILENAME}.F90" "-flag1 -flag2 ...")
endif() |
Permanently disable optimizations for a specific file
This is a fairly common occurrence. We have a number of files in E3SM that would take a very long time to compile with optimizations on. You'll need to edit E3SM/components/cmake/build_model.cmake (if you want to deoptimize a file universally (on all platforms)) or in the Depends files (if you want to deoptimize for specific platforms) as follows:
| Code Block |
|---|
if (COMP_NAME STREQUAL "name-of-component")
list(APPEND NOOPT_FILES "file1" "file2" ...)
endif() |
Permanently change compile flags for a specific file for only specific compilers/machines for E3SM component source files
The three actions above can also be made to E3SM/cime/config/e3sm/machines/Depends.$compiler.$machine.cmake in order to only impact that compiler/machine. Note, there can be some confusion here because the Depends.* filename can have a compiler name, a machine name, or both. All matching Depends.* files will be loaded for a build. The order in which these Depends files are processed is: Depends.$compiler, Depends.$machine, Depends.$compiler.$machine.
Permanent change compiler flags for sharedlib source files
Our sharedlib builds still use the class Makefile-based system to drive the builds for most of our sharedlibs. If machine/compiler-specific flag tweaks are needed, you can use the same Depends.* system described immediately above, except be sure to omit the .cmake suffix for the filename and use Makefile syntax.
Clean my build
'cd $EXEROOT/cmake-bld; make clean'
Totally clean my build
'cd $EXEROOT/cmake-bld; rm -rf *'
Build only a specific component
'cd $EXEROOT/cmake-bld; make $comp_class' e.g ("atm")
Clean only a specific component
'cd $EXEROOT/cmake-bld/cmake/$comp_class; make clean'
Customizing build wrapper actions and timings
As of recent (2020) PRs, the E3SM build system will always wrap compilations and links with the e3sm_compile_wrap.py script. The default version of this script lives in E3SM/cime_config/customize/e3sm_compile_wrap.py and will be copied to all cases in $case/Tools. The one in the case is what actually gets used by the build system and is modifiable by the user at any time. The default behavior of this script is to time add timing info for all links and compilations to the e3sm.bldlog. A handy command for listing actions from cheapest to most expensive: `grep "built in" e3sm.bldlog.* | sort -n -k5,5` . This makes it very easy to find expensive actions.
Using SourceMods
The SourceMods mechanism is a way of bundling code changes within a case. We only recommend using SourceMods for small, short-term changes that just apply to one or two cases. For larger or longer-term changes, including gradual, incremental changes towards a final solution, we highly recommend making changes in the main source tree, leveraging version control (git). The SourceMods directory lives under the case and has subdirectories for each active component, example $case/SourceMods/src.eam. For any source file $name.$suffix in the source mod dir for a component, that file will take "precedence" over files of the same name in repo for that component. Our build system will automatically detect SourceMods changes and will know it needs to reconfigure (recreate cache files) when changes to SourceMods are made.
CSM_Share
This code lives in E3SM/share and is built during the sharedlibs build. It has a cmake build system very similar to one of our core components. Depends files for csm_share live in E3SM/share and need to be kept separate from the "main" Depends files.
Using custom TPLs/sharedlibs
Developers can force the build system to use custom installations of E3SM dependencies by making changes to ${PACKAGE_NAME}_ROOT in their shell environment. As an example, if you set Trilinos_ROOT, your build will build against that Trilinos.