...
- E3SM/cime_config/machines/config_machines.xml: Contains list of valid compilers (COMPILERS) and environment setup (modules, environment_variables).
- E3SM/cime_config/machines/config_compilers.xml: DEPRECATED. Contains specifications (flags, link flags, exe names) for compilers (gcc, pgi) as well as machine-specific modifications for those compilers. This system can be reactivated by setting CIME_NO_CMAKE_MACRO=1 in your environment.
- E3SM/cime_config/machines/cmake_macros/*: Contains specifications (flags, link flags, exe names) for compilers (gcc, pgi) as well as machine-specific modifications for those compilers. Replaces config_compilers.xml.
- E3SM/cime_config/machines/Depends.$compiler.$machine.cmake: Contains machine/compiler-specific flag changes for specific E3SM component files
- E3SM/cime_config/machines/Depends.$compiler.$machine: Contains machine/compiler-specific flag changes for shared-lib files, this system still uses Makefile syntax
- E3SM/cime/scripts/Tools/e3sm_compile_wrap.py: The default compile/link wrapper. It is used to produce timings of all operations
- E3SM/cime/scripts/machines/scripts/*: Python scripts to assist in managing cmake macro files.
- E3SM/components/CMakeLists.txt: The root CMakeLists.txt file for E3SM, mostly just iterates over components and calls functions
- E3SM/components/cmake/cmake_util.cmake: Contains some generally useful functions for cmake
- E3SM/components/cmake/build_mpas_model.cmake: Implements the handoff between E3SM's CMake system to MPAS's Cmake system
E3SM/components/cmake/common_setup.cmake: Cmake setup common to all core components, this is only invoked ones (AFTER mpas is configured)
- E3SM/components/cmake/build_model.cmake: Implements how an individual core component gets configures (involves reading the Filepath and CCSM_cppdefs, globbing files, etc)
- E3SM/components/cmake/modules/*.cmake: Cmake modules for finding packages we depend on
- E3SM/components/cmake/*/CMakeLists.txt: Very thin CMakeLists.txt files for each component type, this is just to ensure each component get it's own build area
- E3SM/components/*/bld/configure: Perl configure script for component, should produce Filepath and CCSM_cppdefs files
- E3SM/components/*/cime_config/buildnml: The buildnml script for component, controls what flags get passed to $component/bld/configure
- E3SM/components/*/cime_config/buildlib: The legacy buildlib driver for the classic deprecated Makefile-based build system
- E3SM/components/*/cime_config/buildlib_cmake: A insertion point for last-minute, component-specific changes before cmake is called. For example, mosart uses this file, not bld/configure, to generate Filepath and CCSM_cppdefs
- E3SM/components/mpas-source/src/CMakeLists.txt: The root CMakeLists.txt for MPAS
- E3SM/components/mpas-source/src/cmake_utils.cmake: Contains some generally useful functions for configuring MPAS
- E3SM/components/mpas-source/src/build_core.cmake: A generic implementation for building MPAS cores
- E3SM/components/mpas-source/src/core_landice/landice.cmake: The CMake definition of the landice core
- E3SM/components/mpas-source/src/core_ocean/ocean.cmake: The CMake definition of the ocean core
- E3SM/components/mpas-source/src/core_seaice/seaice.cmake: The CMake definition of the seaice core
- $case/Buildconf/*conf: The various build conf dirs for all active components, the Filepath and CCSM_cppdefs files for each component live here
- $case/Macros.cmake: The file that includes the relevant files in cmake_macros. You can modify this locally in your case.
- $case/cmake_macros/*: The files that ultimately sets flags, link flags, etc. You can modify this locally in your case.
- $case/Macros.make: A makefile macro derived from the cmake macros, needed for the sharedlibs that still use Makefile build systems. This is an auto-generated file that you should never edit.
- $case/SourceMods/src.$component/*: Alternative source files to use instead of the ones from the component in your source dir
- $EXEROOT/cmake-bld: The root cmake build/binary directory
- $case/Tools/e3sm_compile_wrap.py: The case-specific compiler wrapper, this will start out as a copy of the e3sm_compile_wrap.py from the repo. Users can customize this one.
...
| Code Block |
|---|
% ./create_test $test_case --no-build # Could also do create_newcase here instead
% cd $case_dir
% ./case.setup # only needed if you did create_newcase instead of create_test
% . .env_mach_specific.sh # set your shell env to match E3SM machine/compiler env
% ./case.build --sharedlib-only # make sure sharedlibs are built, these are not yet handled by cmake
% ./case.build --model-only --dry-run
...
* look for CMake cmd, you can also find this command near the top of the e3sm.bldlog for cases that have already run through the build phase*
...
% cd $exeroot/cmake-bld
% * copy/paste cmake cmd and run * |
...
| Code Block |
|---|
% edit $case/cmake_macros/$compiler_$mach.cmake * Add cmake code to check component name and make the corresponding change, e.g. if (COMP_NAME STREQUAL eam) string(APPEND FFLAGSCMAKE_Fortran_FLAGS " -vector0") endif() % make |
...
- universal.cmake : applies changes globally
- $compiler.cmake : applies changes for a compiler type (on all machines)
- $os.cmake : applies change for an OS
- $mach.cmake : applies changes for a machine (for all compilers)
- $compiler_$os.cmake : applies changes for a compiler/os combo (for all machines)
- $compiler_$mach.cmake : applies changes to a compiler/mach comb
- post_process.cmake: finishes setting up cmake stuff, developers should not edit this file
You can write any cmake you want in these. The macros have access to all the key case settings which can be used for conditionals. The common variables to set/adjust in $compilers_$mach.cmake are :
...
similiar to what any CMake project would use, the only difference is that we don't set these in the CACHE (allows each component to customize). You can set any CMake variable in these macros that you want, for example:
- Language compilation flags CMAKE_<LANG>_FLAGS[_<BUILD_TYPE>], examples:
- CMAKE_C_FLAGS: flags for compiling C source files
- CMAKE_C_FLAGS_DEBUG: flags for compiling C sources files for DEBUG builds
- CMAKE_Fortran_FLAGS: flags for compiler fortran source files
- CMAKE_CXX_FLAGS: flags for compiling CXX source files
CMAKE_EXE_LINKER_FLAGS: link flags
- There are tons of CMake variables available to tweak your builds: https://cmake.org/cmake/help/latest/manual/cmake-variables.7.html
- There are a number of variables that are specific to E3SM that are also available:
- CPPDEFS: c preprocessor definitions for all builds.
- CPPDEFS_DEBUG: c preprocessor definitions added for DEBUG buids.
- The serial/mpi compiler names (SCC, SCXX, SFC, MPICC, MPICXX, MPIFC). We still need these (instead of just using CMAKE_<LANG>_COMPILER ) because some of our build system needs to know about serial vs. MPI compilers
SPIO_CMAKE_OPTS: cmake options to pass along to your scorpio build
PIO_FILESYSTEM_HINTS: this setting also gets passed to scorpio
- USE_HIP / USE_CUDA: enable gpu
E3SM_LINK_WITH_FORTRAN: tells cmake to use the fortran compiler to link the e3sm.exe. Some languages must use fortran to link a program that has a fortran main function
- CONFIG_ARGS: autoconf configure arguments to pass to sharedlibs that use autoconf (MCT and mpi-serial)
KOKKOS_OPTIONS: cmake options to pass along to your kokkos build
For the first time, our build system supports conditionals based on versions (instead of having to make entirely new compiler definitions). Example:
| Code Block |
|---|
if (CMAKE_Fortran_COMPILER_VERSION VERSION_GREATER_EQUAL 10) string(APPEND FFLAGSCMAKE_Fortran_FLAGS " -fallow-argument-mismatch -fallow-invalid-boz ") endif() |
...
| Code Block |
|---|
if (COMP_NAME STREQUAL "name-of-component")
e3sm_add_flags("${SOURCE_FILE}" "-flag1 -flag2 ...")
endif() |
You'll need to refer to the source file via it's relative path relative to E3SM/components, e.g. cam/src/physics/rrtmg/ext/rrtmg_lw/rrtmg_lw_k_g.f90
...
) |
You'll need to edit refer to the source file via it's relative path relative to E3SM/components/cmake/build_model.cmake (or the appropriate Depends file) as follows:
...
, e.g. cam/src/physics/rrtmg/ext/rrtmg_lw/rrtmg_lw_k_g.f90
Permanently change compile flags for a generated file (.F90.in)
...
This is a fairly common occurrence. We have a number of files in E3SM that would take a very long time to compile with optimizations on. You'll need to edit E3SM/components/cmake/build_model.cmake (if you want to deoptimize a file universally (on all platforms)) or in the Depends files (if you want to deoptimize for specific platforms) as follows:
| Code Block |
|---|
if (COMP_NAME STREQUAL "name-of-component") list(APPEND NOOPT_FILES "file1" "file2" ...) endif() |
...