Integrator Guide

Owned by Robert Jacob
Last updated: Apr 15, 2024Version comment
15 min read

Integrator Mailing List

All integrators should be subscribed to the integration email list:   e3sm-integrators@lists.mcs.anl.gov  (Contact Robert Jacob to be added).  This list has a daily posting as to the status of the 'next' and 'master' branches with regards to scheduling pull requests.      

Integrator Roles

The Integrator's role is to take a finished feature branch and integrate it into master, with some care.  The developer of the feature branch assists the integrator.

E3SM Integrators (and github username)

Atmosphere:  Balwinder Singh (singhbalwinder), Wuyin Lin (wlin7), Aaron Donahue

Atmosphere MMF:  Walter Hannah (whannah1), Ben Hillman (brhillman), Christopher Jones (Unlicensed) (crjones-amath)

HOMME:  Mark Taylor (mt5555), Oksana Guba (oksanaguba)

Land/River: Gautam Bisht (bishtgautam), Peter Schwartz (peterdschwartz)

MPAS-O/LI/SI: Jon Wolfe (jonbob), Matt Hoffman (matthewhoffman)

   (For MPAS bug reporting, assign bugs as follows:   MPAS-O:  Mark Petersen,  MPAS-Sea ice:  Adrian Turner,  MPAS-Land Ice: Matt Hoffman,  Any other MPAS: Jon Wolfe)

Drv/share/utils/dead:  Robert Jacob (rljacob), Jayesh Krishna (jayeshkrishna), Azamat Mametjanov (amametjanov)

CIME:  James Foucar (jgfouca), Robert Jacob

Free agents (can take on any integration task as they see necessary):  James Foucar (jgfouca), Robert Jacob (rljacob)

Machine file integrators (Machine POCs) not already listed above: Youngsung Kim (OLCF), Noel Keen, Danqing Wu, Bibi Mathew, Aaron Donahue (LLNL) (Can integrate any machine-specific changes even if they are the author.)

Integrators know...

  1. The overall development goals for the component and where the proposed check-in fits within those plans and the E3SM development timeline.
  2. How to run the test suite on a platform.
  3. How to conduct a code review.
  4. The E3SM Development Reference
  5. This Integrator Guide.

Integrators can...

  1. Merge code to the master, next or maint permanent branches as appropriate. (Only through pull requests)
  2. Reset next branches.
  3. Create maint branches.
  4. Make tags on the master or maint branches following Branch, Tag, and Version name conventions.
  5. Verify bugs and, working with group leads, assign developers to fix them.

Can I merge my branch?

(TODO:  write more detailed instructions)

The answer to this question can basically be found by asking "given the current state of the dashboard AND what has already been integrated to next today, will I be able to tell on tomorrow's dashboard if my branch passed its testing?"

Integrating a Feature Branch.

Before Integrating a Feature Branch

Configure your local git to never perform fast forward merges (From within your local ACME repo): git config merge.ff false

To keep the testing from confusing results, only ONE non-BFB merge to next or master can be done per day.

NOTE: Later in this guide will be a section about using forks. If you're integrating a branch from a fork, please refer to both this section and the later section.

You will be the "assignee" on a Pull Request from a feature branch Developer (see last step of Development Getting Started Guide) and oversee the testing and code review.  You may re-assign the lead to one of the other Integrators or ask for additional help.  There should only be one assignee.

  1. Make sure the PR title and description follows the standard described in Development Reference#Submittingapullrequest(PR)
    1. EDIT the PR title and description in github as needed or ask the developer to fix it. Work with developer to craft the PR description.
    2. If the developer has not done so, label the pull request with appropriate component labels on GitHub (i.e. scripts, machinefiles, land, atmosphere, etc...).
    3. If the branch is fixing a bug, make sure the issue # for the bug is mentioned in the PR description as "Fixes #NN" and the "bug fix" label has also been applied to the PR
    4. If the developer did not state what testing was run in the comments, ask them.  They should at minimum  run the e3sm_developer test suite on their branch OR run the 2 test cases used for porting if e3sm_developer is not available AND add new tests for new features if appropriate.  You can run the developer test on their branch for them if you wish.
    5. Make sure the BFB (non-BFB, CC) status of the PR is noted in the description and the same github label is applied.
    6. If the PR is updating a submodule, make sure all of the developer's changes have been merged to the submodule repo first: e.g. developer's changes in their fork were merged into main submodule repo.
  2. If you are satisfied the branch has been tested, conduct Phase 3 of the code review.  See Development Reference#IntegratorCodeReview(Phase3of) Your group may have additional requirements for the code to pass review.  You may ask specific people to help review by adding them to the PR with the "Reviewers" pull down menu.
  3. Merge feature branch into "next" branch:  (If the target for the branch is maintenance branch, skip to "Maintenance Branch Integration" below)
    1. Start with your clone of E3SM on a machine you are comfortable developing on.
    2. Checkout the branch you are going to merge.
      1. git checkout author/branch-name    (note that you don't need to include "origin" in the branch-name.  Git will automatically set up a tracking branch.)
    3. Look at the code changes either on github or using:  git log --reverse -p master..
    4. Assuming tests pass, do  "git checkout next"
    5. Update your version of origin/next with "git fetch origin"
    6. Update your local next with  "git reset --hard origin/next"
    7. Do the merge with:  "git merge --no-ff author/branch-name".
    8. If there were no conflicts, edit the commit message to follow the Commit and PR message template.
    9. If there were conflicts, try to resolve them:   If you are unsure how to proceed, ask for help!
      1. List the problem files with 'git status', edit to fix, commit the changes with "git commit -a" which should complete the merge.
      2. Instead of fixing conflicts during the merge, you could ask the developer to rebase the feature branch to the head of master.  Don't merge master in to the branch just to resolve conflicts.  See Development Reference#AvoidRoutineMergesFromMaster
      3. If you can't fix the problems yourself, try working with the developer as outlined in 126846523
    10. Verify the merge was performed correctly:  DAG is as expected, compare the new merge to what was on next before to make sure you are creating the history you think you are, etc...  Options include:  
      1. git log --graph --oneline --decorate next  (add "--no-color" for a dumb terminal)
      2. git log --graph --oneline --decorate next origin/next
      3. git diff  origin/next..next
    11. Run the e3sm_developer test suite on your local merged next before pushing, to verify the merge was done correctly.
    12. If next is open, push the changes to the main repo with "git push origin next"
  4. Wait for automated testing harness to run the Integration Test Suite (nightly) on next and confirm that all tests still pass.  Results are on http://my.cdash.org/index.php?project=ACME_Climate.
    1. If the integration tests DO NOT pass, determine if the fails/diffs are due to expected baseline differences OR unexpected differences and/or test failures.  Information about the test suites is located on the pages Using the E3SM CDash Dashboard and Interpreting test results.
      1. IF ALL DIFFS ARE EXPECTED (because answers changed or you added/changed a test)
        1. proceed to step 5 and merge to master. 
        2. On the same day you merge to master, file a request to bless the tests at https://acme-climate.atlassian.net/servicedesk/customer/portal/2.
          1. You must merge to master the same day that the next-testing diffs were blessed. This is because we now use the same baselines for next and master.   If you don't merge to master on the same day the baselines are updated, master will report a fail for tests run that night because its using the old code with the new baseline 
          2. merge and then request a bless in that order because it removes the possibility of a bless happening without a merge and for namelist blesses it's helpful to have the PR on master.
      2. For unexpected diffs or test fails.
        1. If a bug is found quickly, fix the bug, commit the fix to the feature-branch, re-merge the branch to next and repeat step 4
        2. If you can not fix the problem quickly, revert the merge commit and work on a fix.  You can then start again at step 3.  NOTE ABOUT REVERTING: Care needs to be taken with any branch that has had a merge reverted on next (or any integration branch).  In a subsequent merge of the same branch (e.g., you merge to next, find a problem, revert the merge to next, add new commits to the branch to fix the issue, merge to next again), git will exclude from the new merge any commits it already sees in the history of next (even though they were subsequently reverted), which can lead to a very confusing 'partial' merge of the updated branch.  There are two possible solutions to deal with this. 1. Revert the revert on next before re-merging the branch to 'reactivate' the reverted commits.  See https://git-scm.com/blog/2010/03/02/undoing-merges.html, "Reverting the revert" for details.  2. rebase the branch so all the hashes change.  This prevents git from matching the commits on the branch to the otherwise identical commits that were previously merged.  (An interactive rebase where you simply change the commit message by a character in the first commit would also satisfy this criterion.)
        3. If the branch is fundamentally flawed, revert the merge commit to next, close the PR (on github) and work with the developer to decide if the branch should be deleted.
    2. If tests DO pass, proceed to step 5.
    3. Testing can only be skipped for a "hot fix" or for code outside the component models IF the code can be verified by inspection to be correct and have no side effects.  Add "Skipping Integration Testing" as a comment in the PR on github.
  5. Merge the pull request to master.
    1. checkout master with "git checkout master"
    2. Update your version of origin/master with "git fetch origin"
    3. Update your local master with "git reset --hard origin/master"
    4. Do the merge with "git merge --no-ff author/branch-name"
    5. If there were no conflicts, edit the commit message to follow the Commit and PR message template.
    6. If there were conflicts, try to resolve them:  If you are unsure how to proceed, ask for help!
      1. List the problem files with 'git status', edit to fix, commit the changes with "git commit -a" which should complete the merge.
      2. If you can't fix the problems yourself, try working with the developer as outlined in 126846523
    7. Verify the merge was performed correctly (in terms of the DAG): git log --graph --oneline --decorate master  (or  'git log --graph --oneline --decorate master --no-color')
    8. If you are 126846523, push the changes to the main repo with "git push origin master"

      1. You may use the github automerge button (the green button) only for the merge to master.  Edit the the title and commit message to match the Commit and PR message template.  The default title does not follow ACME conventions.


  6. If the PR changed answers, either climate changing or roundoff, record details of the PR in the table on Answer-changing commits.
  7. Delete the feature branch after the merge has been completed.   This does not have to be done immediately.  You can use the "Delete branch" button on github.

Maintenance Branch Integration: 

If the target of the feature branch is a maintenance branch (the "base" is set to a maintenance branch) you should not merge the branch to next.  Instead, make sure the developer has tested the branch and integrate it directly to the indicated maintenance branch.

Important notes

Integrators should not merge their own development branches to master without a review from another E3SM staff person.   An exception is made for machine files which must be tested on a specific platform integrators may not have access to.

Integrators should never force push to master.  Force pushing to master requires project wide emails notifying people what what is going on, and how to ensure they don't run into any issues as a side effect.

Next can be force pushed with approval from the Repo Czar.


Processing a bug

If you've been assigned to a bug, follow these steps:

  1. If the description is not complete, work with the bug reporter, through issue comments, to understand/reproduce bug.
  2. Make sure the issue has the "bug" label and at least one component label (atmosphere, land, ocean, etc.).
  3. Add ONE of the following additional labels:  duplicate, won't fix, invalid, confirmed
    1. duplicate:  If the bug is a duplicate of another bug, mention the issue number and close the issue.
    2. won't fix:  If the bug is really a user error or not a bug in the ACME source.
    3. invalid:  For issues that shouldn't be in our issues section.
    4. confirmed:  The bug is real.
  4. Optional:  Add additional label "Critical" if the bug needs extra attention.  Add label "Minor" if bug fix can be delayed.
  5. Assign the bug to a component developer to fix, using their github account name and the "Assignee" menu on the github issue.  Work with group leads if necessary.
  6. Optional:  After consulting with group leads, create a JIRA task in your group for fixing the bug.  Use the task id in related commit messages.  See Commit and PR message template and /wiki/spaces/Docs/pages/14385182

Reseting Next

When a new tag is created on master, the next branch should be reset. In order to reset the next branch, an integrator should perform the following steps.

  1. Ensure they have the new tag: git fetch origin -p
  2. Once they have the new tag, they can reset their local copy of next to the tag
    1. git checkout next
    2. git reset --hard <tag> (e.g. git reset --hard v0.3)
  3. After next has locally been reset. They can force push next to origin: git push origin +next

Making a Maintenance Branch

When a new tag is created on master, a new maint branch should be created for that tag. An integrator should perform the following steps to create a new maint branch.

  1. Create a new local maint branch off of that tag: git checkout -b maint-<tag> <tag> (e.g. git checkout -b maint-0.3 v0.3)
  2. Push the maint branch: git push origin maint-<tag>

Merging Maint to Master

If a maintenance branch received a new feature that also needs to be on master (such as a bug fix), you can merge the maintenance branch to master after the feature has been merged to the maintenance branch.   If this is not possible because the master branch has had to many changes, have the developer issue a second PR to master. 

Creating a Tag on Master or Maint branches

Tags that are created on master or a maint branch should always be annotate tags. Tags are created on specific commits and don't track branches. In order to create a new tag, an integrator should perform the following steps:

  1. Determine the hash of the commit that should be tagged (this could be a branch name, if the HEAD of the branch should be tagged)
  2. Create a new annotated tag on that commit: git tag -a <tagname> <hash> (again, <hash> can be replaced with a branch name to tag the HEAD of the branch)
  3. An editor will come up to write an annotation for the tag. The tag annotation should describe what the tag is for, and preferably a short changelog of what went into the tag relative to the last tag.
    1. In order to summarize the work that went into the tag, an integrator can use the following command: git log --oneline --decorate --first-parent --graph <last_tag>...<hash> (e.g. git log --oneline --decorate --first-parent --graph v0.2...origin/master)
    2. The summary created with the above command shouldn't be placed directly in the annotation. Instead it should be paraphrased to give an idea of the modifications that went into the model.
  4. Push the newly created tag: git push origin <tagname>

Additional Information

Role of integration branches.

It is important for integrators to understand the role of the integration branches:

  • maint - maintenance branch for bug fixes and portability updates for ACME releases
  • master - stable platform for new development and integration of features for upcoming release
  • next - latest changes for testing, may be unstable and is never merged to 'master' or 'maint'

An additional purpose of next is to help weed out features and bugs before they land on the master branch. The master branch is intended to be as stable and bug free as possible. As a result, we use the next branch to try and help enforce this. Some use cases the emphasize the benefit next provides:

  1. A developer implements a new feature that passes the developer test suite, but serious issues are brought to light when the feature is pushed onto next. In the end, it is decided that the feature needs to be removed and re-designed from scratch.
    1. In this case, the merge onto next would be reverted, the feature branch can be deleted, and all can be forgotten.
    2. A benefit this workflow provides, is that the feature that causes major problems never lands on master, so master is unaffected by these major issues.
  2. A developer implements a new feature that bases the developer test suite, and additional bugs are showcased when the integration test suite is run. In this case, the branch can have additional commits which fix the bugs added onto it, and the branch can be merged again into next (note, the first merge in next is not reverted in this case).
    1. Here, a feature branch can have multiple merges into next, to make sure we are comfortable calling the feature "stable"
    2. Note, the feature still only has a single merge into master.

The process is summarized in this diagram:

In addition to understanding the typical workflow, an integrator should understand the tagging policies defined in Branch, Tag, and Version name conventions

Integrating from a fork

As developers can choose to work in a fork of E3SM-Project/E3SM it's important for all integrators to know what a fork is, and understand how to integrate from a fork.

A fork is nothing more than a developers personal clone of E3SM-Project/E3SM that is hosted on github. A fork will be usually be created under a user account, although an organization can also create a fork. A fork of E3SM-Project/E3SM will result in a repository named username/E3SM (e.g. douglasjacobsen/E3SM).

When a developer works in a fork, their branches exist in their fork, and not in E3SM-Project/E3SM.

To integrate a branch from a fork to the E3SM master, you need  to gain access to the fork's branches.  To do this you need to add a new remote to your local repository that points to the developer's fork. For example:

git remote add username/E3SM git@github.com:username/E3SM.git

use "git remote -v" to see the remotes your clone is tracking.

While replacing username with the developers actual github username will add a new remote to your local repository named username/E3SM. Once the remote is added, you can pull down that remote's remote tracking branches using:

git fetch username/E3SM -p

The -p flag will prune any removed branches in addition to update other branches.

After all remote branches are fetched, they will show up when using `git branch -a` or `git branch -r` namespaced under username/E3SM.

Typical integration can now continue as listed above, using the remote pointing to the developers fork in places of origin.

Example of a successful merge and push to master

This is what you might see if you there are no conflicts.

blogin3[100]: git checkout master
Switched to branch 'master'


blogin3[101]: git branch
* master
  next
  rljacob/machines/add-anl-cluster


blogin3[102]: git pull
Already up-to-date.


blogin3[103]: git merge rljacob/machines/add-anl-cluster
Auto-merging scripts/ccsm_utils/Machines/config_machines.xml
Merge made by the 'recursive' strategy.
 scripts/ccsm_utils/Machines/config_compilers.xml    | 24 ++++++++++++++++++++++++
 scripts/ccsm_utils/Machines/config_machines.xml     | 20 +++++++++++++++++++-
 scripts/ccsm_utils/Machines/config_pes.xml          | 16 +++++++++++-----
 scripts/ccsm_utils/Machines/env_mach_specific.blues | 25 +++++++++++++++++++++++++
 scripts/ccsm_utils/Machines/mkbatch.blues           | 85 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 164 insertions(+), 6 deletions(-)
 create mode 100755 scripts/ccsm_utils/Machines/env_mach_specific.blues
 create mode 100755 scripts/ccsm_utils/Machines/mkbatch.blues

blogin3[104]: git push origin master
Counting objects: 16, done.
Delta compression using up to 16 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 934 bytes, done.
Total 6 (delta 5), reused 0 (delta 0)
To git@github.com:ACME-Climate/ACME.git
   c355863..8fdf993  master -> master

Resolving merge conflicts

Sometimes when attempting to integrate a feature, an integrator might encounter several merge conflicts they don't know how to deal with. In order to make the job of an integrator easier, they can employ the help of the developer who submitted the pull request using the following steps:

The integrator can request the developer:

  • Fetch the history of ACME-Climate/ACME: git fetch origin
  • Create a new branch at the head of their current branch with the same name but -resolved appended to the end: 
    • git branch user/component/branch-resolved
  • Merge the target branch into this new branch: 
    • git checkout user/component/branch-resolved; 
    • git merge master (If master is the target. Other targets could be next, maint, or other branches)
  • Resolve the merge conflicts: git status; vim file; git add file; git commit)
    • Another method of resolving merge conflicts is to use git mergetool
    • The commit message for this resolved version is not required to have any specific format.
  • Push the resolved merge commit onto github: git push origin user/component/branch-resolved

After the resolved version is pushed onto github, the developers job is complete. Now the integrator can attempt the merge again, using the resolved branch to fix any merge conflicts.

  • Fetch the history of ACME-Climate/ACME: git fetch origin
  • Checkout the target branch: git checkout master (If master is the target. Other targets could be next, maint, or other branches)
  • Merge topic branch into target branch: git merge origin/user/component/branch (NOTE: Don't merge the -resolved version)
  • Resolve merge conflicts using the -resolved branch: git checkout origin/user/component/branch-resolved -- .
  • Finish merge commit, and edit to follow the format described in Commit and PR message template: git commit
  • Push merge commit onto target branch in ACME-Climate/ACME: git push origin master


Racy Integration

Two people occasionally attempt to merge at about the same time, in which someone will "lose the race". It usually goes like this: you checkout 'next', pull to make sure you have everything from upstream (you didn't forget this, right?), merge 'my/topic-branch', test, and attempt to push, getting an error like:

To git@github.com/E3SM-Project/E3SM
! [rejected] next -> next (fetch first)
error: failed to push some refs to 'git@github.com/E3SM-Project/E3SM'
hint: Updates were rejected because the remote contains work that you do
hint: not have locally. This is usually caused by another repository pushing
hint: to the same ref. You may want to first merge the remote changes (e.g.,
hint: 'git pull') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

Do NOT perform a non-fast-forward pull on an integration branch. (Doing so creates messy history that [does not summarize nicely](http://git-blame.blogspot.com/2013/09/fun-with-first-parent-history.html) with `git log --first-parent`.) Instead, you have two choices. The cleanest is to gracefully lose the race and merge again.

$ git reset --hard origin/next
$ git merge my/topic-branch
... build and test ...
$ git push

This produces clean history with no evidence that you encountered a race and had to try again. If your merge had significant conflicts or if the testing you just did was especially onerous, you can switch hats and merge the result:

$ git reset --hard origin/next
$ git merge ORIG_HEAD
# edit commit message to state which branch was actually merged
# due to losing a race to the integration branch.
... build and test ...
$ git push

This keeps both merge commits (which is a record that there was a race, or just clutter) but `git log --first-parent` still produces an accurate and concise summary.

Bypassing hooks

Sometimes when integrating a branch, the branch name is really long which triggers an error using our ACME-Hooks. In this case, the hook can be bypassed after ensuring that your commit message looks correct by issuing the following command:

git commit --no-verify


Integrating machine file changes

If the changes are only to machine files:

Depends.$machine
env_mach_specific.$machine
mkbatch.$machine
syslog.$machine

or the machine-specific blocks config_machines.xml, config_compiler.xml or testlist.xml then the POC of the machine (see Configuration Management) can integrate them even if they are the author of the branch.