B5: Port iESM to E3SM Design Document

Owned by Renata McCoy
Last updated: Sept 22, 2020 by Katherine Calvin (Unlicensed)
7 min read

The Design Document page provides a description of the algorithms, implementation and planned testing including unit, verification, validation and performance testing. Please read  Step 1.3 Performance Expectations that explains feature documentation requirements from the performance group point of view. 

Design Document

 Click here for instructions to fill up the table below ......

The first table in Design Document gives overview of this document, from this info the Design Documents Overview page is automatically created.

In the overview table below 4.Equ means Equations and Algorithms, 5.Ver means Verification, 6.Perf - Performance, 7. Val - Validation

  • Equations: Document the equations that are being solved and describe algorithms
  • Verification Plans: Define tests that will be run to show that implementation is correct and robust. Involve unit tests to cover range of inputs as well as benchmarks.
  • Performance expectations: Explain the expected performance impact from this development
  • Validation Plans: Document what process-based, stand-alone component, and coupled model runs will be performed, and with what metrics will be used to assess validity

Use the symbols below (copy and paste) to indicate if the section is in progress or done or not started.

In the table below 4.Equ means Equations and Algorithms, 5.Ver means Verification, 6.Perf - Performance, 7. Val - Validation,   (tick) - competed, (warning) - in progress, (error) - not done


Overview table for the owner and an approver of this feature

1.Description

Port iESM Code to E3SM
2.Owner

Katherine Calvin (Unlicensed)

3.Created

 

4.Equ
5.Ver
6.Perf
7.Val
8.Approver
 
9.Approved Date
V2.0Pending
 Click here for Table of Contents ...

Table of Contents




Title: B5: Port iESM to E3SM

Requirements and Design

E3SM Land Group

Date:     

Summary

This design document covers the port of the iESM code (https://github.com/E3SM-Project/iESM) to the E3SM. The modifications in this document are summarized as follows:
  • Include GCAM in the E3SM code base
  • Call the GCAM component as part of the coupler timing scheme
  • Pass information from GCAM to ELM on LULCC
  • Pass information from GCAM to EAM on CO2 emissions
  • Pass information from ELM to GCAM on NPP and HR


Requirements

Requirement: Replicate the iESM science in E3SM 

Date last modified: 
Contributors: Katherine Calvin (Unlicensed)

The iESM links GCAM to CESM, with the coupling primarily focused on the carbon cycle. Fossil fuel & industrial CO2 emissions AND land use/land cover change are passed from GCAM to the Earth System. Net primary productivity and heterotrophic respiration are passed back to GCAM from the land model. The exchange of information occurs every five years (the native GCAM time step). Data is aggregated/disaggregated between models (GCAM operates at regional scale and 5-year time steps; the ESM operates on a grid with 30 minute time step). Code is documented in Collins et al. (2015); a science example is provided in Thornton et al. (2017). The full iESM repository is available at: https://github.com/E3SM-Project/iESM

Collins, W. D., A. P. Craig, J. E. Truesdale, A. V. Di Vittorio, A. D. Jones, B. Bond-Lamberty, K. V. Calvin, J. A. Edmonds, S. H. Kim, A. M. Thomson, P. Patel, Y. Zhou, J. Mao, X. Shi, P. E. Thornton, L. P. Chini and G. C. Hurtt (2015). "The integrated Earth system model version 1: formulation and functionality." Geosci. Model Dev. 8(7): 2203-2219.

Thornton, P. E., K. Calvin, A. D. Jones, A. V. D. Vittorio, B. Bond-Lamberty, L. Chini, X. Shi, J. Mao, W. D. Collins, J. Edmonds, A. Thomson, J. Truesdale, A. Craig, M. L. Branstetter and G. Hurtt (2017). "Biospheric feedback effects in a synchronously coupled model of human and Earth systems." Nature Clim. Change 7: 496-500.

Requirement: Ensure that all possible configurations of E3SM & GCAM can run

Date last modified: 

Contributors: Katherine Calvin (Unlicensed)

GCAM is an option that we will only need for some simulations. Additionally, we may want to run GCAM-ELM or GCAM-EAM, in addition to GCAM with the fully coupled model. We need to ensure that GCAM can be turned on and off like other components of E3SM through the definition of compsets.


Requirement: Ensure the GCAM couplings can be turned on and off

Date last modified: 

Contributors: Katherine Calvin (Unlicensed)

We will need to be able to flexibly control which information is passed between GCAM and E3SM so that we can quantify the effect of these couplings. We need to ensure that the information passed to GCAM can be controlled via compsets or namelists.

Requirement: Implement the GCAM-E3SM coupling in a manner that can be extended in the future

Date last modified: 

Contributors: Katherine Calvin (Unlicensed)

The phase 2 proposal includes extensions to the coupling between E3SM and GCAM. Additionally, GCAM is continually developing and we need to ensure we can incorporate those changes. Specifically, we will want to be able to adapt the coupling time step, the regional mappings, and which variables are passed. We need to ensure that the system is flexible and extensible.

Algorithmic Formulations, and Design Implementation



Design Solution: Port the iESM coupling code to E3SM and Link GCAM to E3SM via the coupler

Date last modified: 
Contributors: Tim Shippert (Unlicensed), Katherine Calvin (Unlicensed)

Including the IAC component

Submodules: GCAM, and the other parts of the IAC code, are added via submodule. To minimize the amount of code reorganization, we technically have two submodules. The e3sm-integration branch of the main GCAM code (https://github.com/JGCRI/gcam-core) is included as a submodule to the https://github.com/E3SM-Project/giac repository, which is then submoduled into E3SM. giac includes some coupling code, in particular, the code to downscale, harmonize, and create land surface files. Ideally, we would include all of the relevant GCAM post-processing and coupling code in a single repository, but that isn't practical right now so we are using nested submodules.


IAC component implementation

  1. Coupler modifications
    1. cesm_comp_mod.F90
      1. Set up iac vars,  comp type, etc.
      2. modify general component function calls to include iac component
        1. … list function calls
        2. prep_lnd_init
        3. prep_atm_init
      1. RECV/run/SEND sections for iac component
      2. modify general component function calls to include iac component
        1. … list function calls
      1. Header: include iac modules and vars
      2. Mod cesm_init() function
      3. Mod cesm_run() function
  2. New iac coupler interface
    1. iac_comp_mct.F90
      1. iac_init_mct()
      2. iac_run_mct()
      3. iac_final_mct()
      1. New functions
    2. New files

<INSERT FIGURE SHOWING HOW THE IAC COUPLING WORKS>

Changes to the coupler

USE_GCAM flag added. If true, different compile flags are set.

cime/src/drivers/mct/main/cime_comp_mod.F90:

  • iac_c2_atm added to the atm initialization
  • prep calls are added to atm if iac_present is true
  • prep calls are added to land if iac_c2_lnd exists
  • An iac clock Eclock_z is added
  • prep_iac_mrg added

cime/src/drivers/mct/main/prep_atm_mod.F90:

  • iac or component "z" added wherever components are listed, used, defined, etc.

cime/src/drivers/mct/main/prep_iac_mod.F90: 

  • iac prep function expanded to include logging, checks on the iac and land
  • Includes accumulator functions to aggregate data from the land to the iac temporal resolution. 
    • Takes a max of monthly means – the max gets the end of the growing season when NPP is highest, the mean over the month reduces noise prior to passing the data to iac
  • Implements iac_prep_mrg

cime/src/drivers/mct/main/prep_lnd_mod.F90:

  • iac or component "z" added wherever components are listed, used, defined, etc.

KATE TO CHECK: currently code assumes iac is on lnd grid. See if that is changed in final version. If not, need to document clearly here (and update GCAM mapping files whenever land grid changes)

cime/src/drivers/mct/shr/seq_flds_mod.F90:

  • iac or component "z" added wherever components are listed, used, defined, etc.
  • Adds coupling code for lnd->iac, iac->lnd, iac→atm
    • lnd → iac
      • Defines the state variables that are passed from land to iac (NPP, heterotrophic respiration, PFT weight)
      • Each variable is passed for each PFTs 
      • KATE: it says four variables but I only see 3 in code - I think we need to pass % vegetated land too?
    • iac → lnd
      • Defines the single state variable passed from iac to land (percent of vegetated land unit for each PFT)
    • iac → atm
      • Defines the single flux variable passed from iac to atm (surface flux of CO2)
      • KATE will need to modify after aircraft and monthly is added, depending on how this is done


  1. Initialization and communications: modify modules and functions to include iac interface variables and perform iac initializations
    1. IACID, ALLIACID, CPLALLIACID, CPLIACID, numinst_iac
    2. Modify functions to work with iac
    1. seq_infodata_getData, seq_infodata_putData, seq_infodata_init, other functions
      1. iac_present, iac_prognostic, iac_phase, others
      2. explicit and/or bytype interface?
    2. seq_comm_mct.F90
    3. seq_infodata_mod.F90 - add iac

Changes to ELM

components/clm/src/cpl/clm_cpl_indices.F90:

  • NPP, HR, and PFT weights added as coupling indices (these are the variables passed from land to iac)
    • Indices are also accumulated

components/clm/src/cpl/lnd_comp_mct.F90:

  • lnd2iac_vars added

components/clm/src/cpl/lnd_import_export.F90:

  • lnd2iac_vars added to lnd_export

components/clm/src/main/clm_driver.F90:

  • Add call to lnd2iac if iac_active is true

components/clm/src/main/clm_iniatilizeMod.F90:

  • Add call to init_lnd2iac if iac_active is true

components/clm/src/main/clm_instMod.F90:

  • Initialize lnd2iac
  • NOTE: there are calls to initialize iac2lnd, but they are currently commented out.

components/clm/src/main/clm_varctl.F90:

  • iac_active added as a control variable, defaulted to false

components/clm/src/main/iac2lndMod.F90:

  • New File!
  • Defines Init, Restart, and update_iac2lnd subroutines
    • Init allocates memory for variables
    • Restart reads and writes iac2lnd information to/from the restart file
    • update_iac2lnd sets the ELM variables from the information passed from iac
      • Loops over the 1D array from iac and extracts the information for each patch and PFT on this grid

components/clm/src/main/lnd2iacMod.F90:

  • New File!
  • Defines Init and update_lnd2iac subroutines
    • KATE: Why no Restart?
    • Init allocates memory for hr, npp, and pftwt for all grid x PFT combinations
    • update_lnd2iac loops over patch and extracts variables by pft for this grid


Changes to EAM


components/cam/src/control/camsrfexch.F90: 

  • Input from IAC to EAM defined (fco2_iac)

components/cam/src/cpl/atm_import_export.F90: 

  • Code modified to check if iac component passed data first. If not, read from file

components/cam/src/cpl/cam_cpl_indices.F90: 

  • co2 flux from iac added to list of co2 flux indices


Design Solution: Implement compsets to activate different combinations of components

Date last modified: 
Contributors: Katherine Calvin (Unlicensed)

At this point, we have added the following compsets:

<INSERT NAME>: Couples GCAM and ELM only (effectively an I compset + GCAM)

<INSERT NAME>: Couples GCAM, EAM, and ELM (effectively an F compset + GCAM)

Note that we will subsequently add different compsets for different future forcing scenarios (e.g., different RCPs). The compsets listed above are used to demonstrate capabilities and for testing.


Design Solution: Enable couplings through namelist options

Date last modified: 
Contributors: Katherine Calvin (Unlicensed)

The following namelist variables have been added:

KATE TO ADD LIST


Design Solution: Embed algorithms for temporal & spatial resolution changes within the IAC component that are as flexible as possible

Date last modified: 
Contributors: 

.....description.....

.....reference.....

 

Planned Verification and Unit Testing 

Verification and Unit Testing: Expert Review

Date last modified: 
Contributors: 

The named experts provided extensive review of each other's work, to verify that the design and implementations met the stated requirements. Model testing at each stage of implementation was used to verify that intended changes were correctly implemented, and that unintended changes were avoided.


Planned Validation Testing 

Validation Testing:  Evaluation of data exchanged between components

Date last modified: 
Contributors: Katherine Calvin (Unlicensed), Alan Di Vittorio

The results of the ported code will eventually be analyzed in a similar manner to that of the iESM (e.g., Thornton et al., 2017). We will not expect exact matches, since we are using a different ESM, but we can ensure the behavior matches expectations/previous experience. For this code review, we will use a set of short test simulations and examine the inputs, outputs, and logs of all components to ensure that the information is being passed correctly in each configuration. We will document the exact checks and archive the code when possible so this can be replicated later.

Planned Performance Testing 

Performance Testing: Short simulations on multiple platforms

Date last modified: 
Contributors: Katherine Calvin (Unlicensed)

We do not expect GCAM to alter the performance of the coupled system, but we will perform short coupled simulations on multiple platforms to quantify the change.