Integrator Guide

Owned by Robert Jacob
Last updated: about 14 hours agoVersion comment
15 min read

Integrator Mailing List

All integrators should be subscribed to the integration email list:   e3sm-integrators@lists.mcs.anl.gov  (Contact Robert Jacob to be added).  This list has a daily posting as to the status of the 'next' and 'master' branches with regards to scheduling pull requests.      

Integrator Roles

The Integrator's role is to take a finished feature branch and integrate it into master, with some care.  The developer of the feature branch assists the integrator.

E3SM Integrators main area (and github username)

EAM:  Balwinder Singh (singhbalwinder), Wuyin Lin (wlin7)

EAM MMF:  Walter Hannah (whannah1), Ben Hillman (brhillman)

EAMxx: Aaron Donahue (AaronDonahue), Luca Bertagna (bartgol), Conrad Clevenger (tcclevenger), Andrew Bradley (ambrad)

HOMME:  Mark Taylor (mt5555), Oksana Guba (oksanaguba)

ELM/MOSART: Gautam Bisht (bishtgautam), Peter Schwartz (peterdschwartz)

MPAS-O/LI/SI: Jon Wolfe (jonbob), Matt Hoffman (matthewhoffman)

   (For MPAS bug reporting, assign bugs as follows:   MPAS-O:  Mark Petersen,  MPAS-Sea ice:  Adrian Turner,  MPAS-Land Ice: Matt Hoffman,  Any other MPAS: Jon Wolfe)

Drv/share/utils/dead/driver:  Robert Jacob (rljacob), Jayesh Krishna (jayeshkrishna), Azamat Mametjanov (amametjanov)

CIME:  James Foucar (jgfouca), Robert Jacob

Free agents (can take on any integration task as they see necessary):  James Foucar (jgfouca), Robert Jacob (rljacob)

Machine file integrators (Machine POCs):  Youngsung Kim (OLCF), Noel Keen (NERSC), Danqing Wu (LCRC) Bibi Mathew (Compy) Jason Boutte (LLNL) Azamat Mametjanov (ALCF) (Can integrate any machine-specific changes even if they are the author but still requires at least one review)

Integrators know...

  1. The overall development goals for the component and where the proposed PR fits within those plans and the E3SM development timeline.
  2. How to run the test suite on a platform.
  3. How to conduct a code review.
  4. The E3SM Development Reference
  5. This Integrator Guide.

Integrators can...

  1. Merge code to the master, next or maint permanent branches as appropriate. (Only through pull requests)
  2. Reset next branches.
  3. Create maint branches.
  4. Make tags on the master or maint branches following Branch, Tag, and Version name conventions.
  5. Verify bugs and, working with group leads, assign developers to fix them.

Can I merge my branch?

(TODO:  write more detailed instructions)

The answer to this question for merging to next can be determined by asking "given the current state of the dashboard AND what has already been integrated to next today, will I be able to tell on tomorrow's dashboard if this branch passed its testing?"

Integrating a Feature Branch.

Before Integrating a Feature Branch

Configure your local git to never perform fast forward merges (From within your local E3SM repo): git config merge.ff false

To keep the testing from confusing results, only ONE non-BFB merge to next or master can be done per day.

NOTE: Later in this guide will be a section about using forks. If you're integrating a branch from a fork, please refer to both this section and the later section.

You will be the "assignee" on a Pull Request from a feature branch Developer (see last step of Development Getting Started Guide) and oversee the testing and code review.  You may re-assign the lead to one of the other Integrators or ask for additional help.  There should only be one assignee.

  1. Make sure the PR title and description follows the standard described in Development Reference#Submittingapullrequest(PR)
    1. EDIT the PR title and description in github as needed or ask the developer to fix it. Work with developer to craft the PR description.
    2. If the developer has not done so, label the pull request with appropriate component labels on GitHub (i.e. scripts, machinefiles, ELM, EAM, etc...).
    3. If the branch is fixing a bug, make sure the issue # for the bug is mentioned in the PR description as "Fixes #NN" and the "bug fix" label has also been applied to the PR
    4. If the github PR testing fails and the developer did not state what testing was run in the comments, ask them.  They should have ideally run the e3sm_developer test suite on their branch OR run a test that exercises the new code AND add new tests for new features if appropriate.  The developer is responsible for making the github PR testing pass.
    5. Make sure the BFB (non-BFB, CC) status of the PR is noted in the description and the same github label is applied.
    6. If the PR is updating a submodule, make sure all of the developer's changes have been merged to the submodule repo first: e.g. developer's changes in their fork were merged into main submodule repo.
  2. If you are satisfied the branch has been tested, conduct Phase 3 of the code review.  See Development Reference#IntegratorCodeReview(Phase3of) Your group may have additional requirements for the code to pass review.  You may ask specific people to help review by adding them to the PR with the "Reviewers" pull down menu.
  3. Merge feature branch into "next" branch:  (If the target for the branch is maintenance branch, skip to "Maintenance Branch Integration" below)
    1. Start with your clone of E3SM on a machine you are comfortable developing on.
    2. Checkout the branch you are going to merge.
      1. git checkout author/branch-name    (note that you don't need to include "origin" in the branch-name.  Git will automatically set up a tracking branch.)
    3. Look at the code changes either on github or using:  git log --reverse -p master..
    4. Assuming tests pass, do  "git checkout next"
    5. Update your version of origin/next with "git fetch origin"
    6. Update your local next with  "git reset --hard origin/next"
    7. Do the merge at the command line with:  "git merge --no-ff author/branch-name".
    8. If there were no conflicts, edit the commit message to follow the Commit and PR message template.
    9. If there were conflicts, try to resolve them:   If you are unsure how to proceed, ask for help!   DO NOT MERGE FROM MASTER JUST TO FIX CONFLICTS
      1. If there are conflicts when merging to next, there will likely also be conflicts when merging to master.  Use the git rerere cache to make the merge to master easier. 
      2. List the problem files with 'git status', edit each to fix, add each fixed file with "git add filename".  When done, do "git commit -a" and enter a message about the conflicts fixed.
      3. Instead of fixing conflicts during the merge, you could ask the developer to rebase the feature branch to the head of master.  Don't merge master in to the branch just to resolve conflicts.  See Development Reference#AvoidRoutineMergesFromMaster
      4. If you can't fix the problems yourself, try working with the developer as outlined in 126846523
    10. Verify the merge was performed correctly:  DAG is as expected, compare the new merge to what was on next before to make sure you are creating the history you think you are, etc...  Options include:  
      1. git log --graph --oneline --decorate next  (add "--no-color" for a dumb terminal)
      2. git log --graph --oneline --decorate next origin/next
      3. git diff  origin/next..next
    11. If you are unsure, run the e3sm_developer test suite on your local merged next before pushing, to verify the merge was done correctly.
    12. If next is open, push the changes to the main repo with "git push origin next"
  4. Wait for automated testing harness to run the Integration Test Suite overnight on next and confirm that tests have the expected results.  Results are on http://my.cdash.org/index.php?project=E3SM_Climate.
    1. If the integration tests DO NOT pass, determine if the fails/diffs are due to expected baseline differences OR unexpected differences and/or test failures.  Information about the test suites is located on the pages Using the E3SM CDash Dashboard and Interpreting test results.
      1. IF ALL DIFFS ARE EXPECTED (because answers changed or you added/changed a test)
        1. proceed to step 5 and merge to master. 
        2. On the same day you merge to master, file a request to bless the tests at https://acme-climate.atlassian.net/servicedesk/customer/portal/2.
          1. You must merge to master BEFORE asking for tests to be blessed. This is because we now use the same baselines for next and master.  Also, namelist baselines are updated using master. 
      2. For unexpected diffs or test fails.
        1. If a bug is found quickly, fix the bug, commit the fix to the feature-branch, re-merge the branch to next and repeat step 4
        2. If you can not fix the problem quickly, revert the merge commit and work on a fix.  You can then start again at step 3.  NOTE ABOUT REVERTING: Care needs to be taken with any branch that has had a merge reverted on next (or any integration branch).  In a subsequent merge of the same branch (e.g., you merge to next, find a problem, revert the merge to next, add new commits to the branch to fix the issue, merge to next again), git will exclude from the new merge any commits it already sees in the history of next (even though they were subsequently reverted), which can lead to a very confusing 'partial' merge of the updated branch.  There are two possible solutions to deal with this. 1. Revert the revert on next before re-merging the branch to 'reactivate' the reverted commits.  2. rebase the feature branch so all the hashes change.  This prevents git from matching the commits on the branch to the otherwise identical commits that were previously merged.  (An interactive rebase where you simply change the commit message by a character in the first commit would also satisfy this criterion.) 3. Instead of reverting the branch from next, force push next to the previous commit.  You can only do this if nothing else has been merged to next.
        3. If the branch is fundamentally flawed, revert the merge commit to next, close the PR (on github) and ask the developer to reopen when its working.
    2. If tests DO pass, proceed to step 5.
    3. Testing can only be skipped for a "hot fix" OR for code outside the component models IF the code can be verified by inspection to be correct and have no side effects OR if the PR has undergone its own customized testing (as for EAMxx).  Add "Skipping Integration Testing" as a comment in the PR on github.
  5. Merging to master:  
    1. Merge the pull request to master (Command line, with conflicts).
      1. checkout master with "git checkout master"
      2. Update your version of origin/master with "git fetch origin"
      3. Update your local master with "git reset --hard origin/master"
      4. Do the merge with "git merge --no-ff author/branch-name"
      5. If there were no conflicts, edit the commit message to follow the Commit and PR message template.
      6. If there were conflicts, try to resolve them:  If you are unsure how to proceed, ask for help!
        1. List the problem files with 'git status', edit to fix, commit the changes with "git commit -a" which should complete the merge.
        2. If you can't fix the problems yourself, try working with the developer as outlined in 126846523
      7. Verify the merge was performed correctly (in terms of the DAG): git log --graph --oneline --decorate master  (or  'git log --graph --oneline --decorate master --no-color')
      8. If you are 126846523, push the changes to the main repo with "git push origin master"
    2. Merge the pull request to master (Green button) 

      1. You may use the github automerge button (the green button) only for the merge to master AND only if there are no conflicts. Edit the the title and commit message in the text box on github.com to match the Commit and PR message template.  The default title does not follow E3SM conventions.

      2. When you push the green button, you'll see a template like this.

      3. Edit the title to match our conventions and copy and paste the PR description in to the box for the description as below:
      5. Most importantly, the github number is moved to the end of the title, the word 'branch' is added and the branch name is in single quotes.  This makes it look identical to a git merge done with the command line in the git log.
  6. The github repo is set to delete the branch after it is merged.


Maintenance Branch Integration: 

If the target of the feature branch is a maintenance branch (the "base" is set to a maintenance branch) you should not merge the branch to next.  Instead, make sure the developer has tested the branch and integrate it directly to the indicated maintenance branch.   Because we do not have an integration branch for maintenance branches, only merge one PR per day.  The maintenance branches are tested nightly.  Revert the PR from the maint branch if there are issues.

Important notes

Integrators should not merge their own development branches to master without a review from another E3SM staff person.    Ideally, someone else would be the integrator.

Integrators should never force push to master.  This is disabled with a branch protection rule.

Next can be force pushed if it does not disrupt the work of other integrators (who just merged something and want testing to run on it)


Processing a bug

If you've been assigned to a bug, follow these steps:

  1. If the description is not complete, work with the bug reporter, through issue comments, to understand/reproduce bug.
  2. Make sure the issue has the "bug" label and at least one component label (atmosphere, land, ocean, etc.).
  3. Add ONE of the following additional labels:  duplicate, won't fix, invalid, confirmed
    1. duplicate:  If the bug is a duplicate of another bug, mention the issue number of the duplicate and close the issue.
    2. won't fix:  If the bug is really a user error or not a bug in the E3SM source.
    3. invalid:  For issues that shouldn't be in our issues section.
    4. confirmed:  The bug is real.
  4. Optional:  Add additional label "Critical" if the bug needs extra attention.  Add label "Minor" if bug fix can be delayed.
  5. Assign the bug to a component developer to fix, using their github account name and the "Assignee" menu on the github issue.  Work with group leads if necessary to determine who should be assigned.

Reseting Next

When a new tag is created on master, the next branch should be reset. In order to reset the next branch, an integrator should perform the following steps.

  1. Ensure they have the new tag: git fetch origin -p
  2. Once they have the new tag, they can reset their local copy of next to the tag
    1. git checkout next
    2. git reset --hard <tag> (e.g. git reset --hard v0.3)
  3. After next has locally been reset. They can force push next to origin: git push origin +next

Making a Maintenance Branch

When a new major or minor version tag is created on master, a new maint branch should be created to preserve answers for that version. An integrator should perform the following steps to create a new maint branch.

  1. Create a new local maint branch off of that tag or from a hash after the tag if answers are BFB.
  2. Push the maint branch: git push origin maint-x.y

Merging Maint to Master

If a maintenance branch received a new feature that also needs to be on master (such as a bug fix), you can merge the maintenance branch to master after the feature has been merged to the maintenance branch.   If this is not possible because the master branch has had to many changes, have the developer issue a second PR to master. 

Creating a Tag on Master or Maint branches

Tags that are created on master or a maint branch should always be annotated tags. Tags are created on specific commits and don't track branches. In order to create a new tag, an integrator should perform the following steps:

  1. Determine the hash of the commit that should be tagged (this could be a branch name, if the HEAD of the branch should be tagged)
  2. Create a new annotated tag on that commit: git tag -a <tagname> <hash> (again, <hash> can be replaced with a branch name to tag the HEAD of the branch)
  3. An editor will come up to write an annotation for the tag. The tag annotation should describe what the tag is for, and preferably a short changelog of what went into the tag relative to the last tag.   Draft descriptions are prepared in Confluence pagers such as /wiki/spaces/ED/pages/3896770749
  4. Push the newly created tag: git push origin <tagname>

Additional Information

Role of integration branches.

It is important for integrators to understand the role of the integration branches:

  • maint - maintenance branch for bug fixes and portability updates for E3SM releases.   Keeps old versions running on newer hardware and producing the same climate.
  • master - stable platform for new development and integration of features for upcoming release
  • next - latest changes for testing, may be unstable and is never merged to 'master' or 'maint'

An additional purpose of next is to help weed out features and bugs before they land on the master branch. The master branch is intended to be as stable and bug free as possible. As a result, we use the next branch to try and help enforce this. Some use cases the emphasize the benefit next provides:

  1. A developer implements a new feature that passes the developer test suite and their tests, but serious issues are brought to light when the feature is pushed onto next and interacts with other recent changes as seen in github PR testing.
    1. In this case, the merge onto next would be reverted, the feature branch can be rebased and testing and further development done on the rebased branch.  The developer may need to start from scratch.  PR might have to be closed.
    2. The feature that causes major problems never lands on master, so master is unaffected by these major issues.
  2. A developer implements a new feature that passes the developer test suite, and additional bugs are showcased when the integration test suite is run overnight. In this case, the branch can have additional commits which fix the bugs added onto it, and the branch can be merged again into next (note, the first merge in next is not reverted in this case).
    1. Here, a feature branch can have multiple merges into next, to make sure we are comfortable calling the feature "stable"
    2. Note, the feature still only has a single merge into master.

The process is summarized in this diagram:

In addition to understanding the typical workflow, an integrator should understand the tagging policies defined in Branch, Tag, and Version name conventions

Integrating from a fork

Developers can choose to work in a fork of E3SM-Project/E3SM so it's important for all integrators to know what a fork is, and understand how to integrate from a fork.

A fork is nothing more than a developers personal clone of E3SM-Project/E3SM that is hosted on github. A fork will usually be created under a user account, although a github organization can also host a fork. A fork of E3SM-Project/E3SM will result in a repository named username/E3SM (e.g. rljacob/E3SM).

When a developer works in a fork, their branches exist in their fork, and not in E3SM-Project/E3SM.

To integrate a branch from a fork to the E3SM master, you need  to gain access to the fork's branches.  To do this you need to add a new remote to your local clone that points to the developer's fork. For example:

git remote add usernameE3SM git@github.com:username/E3SM.git

use "git remote -v" to see the remotes your clone is tracking.

While replacing username with the developers actual github username will add a new remote to your local repository named username/E3SM. Once the remote is added, you can pull down that remote's remote tracking branches using:

git fetch usernameE3SM -p

The -p flag will prune any removed branches in addition to update other branches.

After all remote branches are fetched, they will show up when using `git branch -a` or `git branch -r` namespaced under usernameE3SM.

Typical integration can now continue as listed above, using the remote pointing to the developers fork in places of origin.  e.g  "git merge usernameE3SM/branchname"

Example of a successful merge and push to master

This is what you might see if you there are no conflicts.

blogin3[100]: git checkout master
Switched to branch 'master'


blogin3[101]: git branch
* master
  next
  rljacob/machines/add-anl-cluster


blogin3[102]: git pull
Already up-to-date.


blogin3[103]: git merge rljacob/machines/add-anl-cluster
Auto-merging scripts/ccsm_utils/Machines/config_machines.xml
Merge made by the 'recursive' strategy.
 scripts/ccsm_utils/Machines/config_compilers.xml    | 24 ++++++++++++++++++++++++
 scripts/ccsm_utils/Machines/config_machines.xml     | 20 +++++++++++++++++++-
 scripts/ccsm_utils/Machines/config_pes.xml          | 16 +++++++++++-----
 scripts/ccsm_utils/Machines/env_mach_specific.blues | 25 +++++++++++++++++++++++++
 scripts/ccsm_utils/Machines/mkbatch.blues           | 85 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 164 insertions(+), 6 deletions(-)
 create mode 100755 scripts/ccsm_utils/Machines/env_mach_specific.blues
 create mode 100755 scripts/ccsm_utils/Machines/mkbatch.blues

blogin3[104]: git push origin master
Counting objects: 16, done.
Delta compression using up to 16 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 934 bytes, done.
Total 6 (delta 5), reused 0 (delta 0)
To git@github.com:E3SM-Project/E3SM.git
   c355863..8fdf993  master -> master

Resolving merge conflicts

Sometimes when attempting to integrate a feature, an integrator might encounter several merge conflicts they don't know how to deal with. In order to make the job of an integrator easier, they can employ the help of the developer who submitted the pull request using the following steps:

DO NOT MERGE FROM MASTER JUST TO FIX A FEW CONFLICTS.

The integrator can request the developer:

  • Fetch the history of E3SM-Project/E3SM: git fetch origin
  • Create a new branch at the head of their current branch with the same name but -resolved appended to the end: 
    • git branch user/component/branch-resolved
  • Merge the target branch into this new branch: 
    • git checkout user/component/branch-resolved; 
    • git merge master (If master is the target. Other targets could be next, maint, or other branches)
  • Resolve the merge conflicts: git status; vim file; git add file; git commit)
    • Another method of resolving merge conflicts is to use git mergetool
    • The commit message for this resolved version is not required to have any specific format.
  • Push the resolved merge commit onto github: git push origin user/component/branch-resolved

After the resolved version is pushed onto github, the developers job is complete. Now the integrator can attempt the merge again, using the resolved branch to fix any merge conflicts.

  • Fetch the history of E3SM-Project/E3SM: git fetch origin
  • Checkout the target branch: git checkout master (If master is the target. Other targets could be next, maint, or other branches)
  • Merge topic branch into target branch: git merge origin/user/component/branch (NOTE: Don't merge the -resolved version)
  • Resolve merge conflicts using the -resolved branch: git checkout origin/user/component/branch-resolved -- .
  • Finish merge commit, and edit to follow the format described in Commit and PR message template: git commit
  • Push merge commit onto target branch in E3SM-Project/E3SM: git push origin master


Racy Integration

Two people occasionally attempt to merge to next at about the same time, in which someone will "lose the race". It usually goes like this: you checkout 'next', pull to make sure you have everything from upstream , merge 'my/topic-branch', do some testing on the new local version of next, and attempt to push, getting an error like:

To git@github.com/E3SM-Project/E3SM
! [rejected] next -> next (fetch first)
error: failed to push some refs to 'git@github.com/E3SM-Project/E3SM'
hint: Updates were rejected because the remote contains work that you do
hint: not have locally. This is usually caused by another repository pushing
hint: to the same ref. You may want to first merge the remote changes (e.g.,
hint: 'git pull') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

Do NOT perform a non-fast-forward pull on an integration branch. (Doing so creates messy history that [does not summarize nicely](http://git-blame.blogspot.com/2013/09/fun-with-first-parent-history.html) with `git log --first-parent`.) Instead, you have two choices. The cleanest is to gracefully lose the race and merge again.

$ git reset --hard origin/next

(start over)
$ git merge my/topic-branch
... build and test ...
$ git push

This produces clean history with no evidence that you encountered a race and had to try again. If your merge had significant conflicts or if the testing you just did was especially onerous, you can switch hats and merge the result:

$ git reset --hard origin/next
$ git merge ORIG_HEAD
# edit commit message to state which branch was actually merged
# due to losing a race to the integration branch.
... build and test ...
$ git push

This keeps both merge commits (which is a record that there was a race, or just clutter) but `git log --first-parent` still produces an accurate and concise summary.

Bypassing hooks

Sometimes when integrating a branch, the branch name is really long which triggers an error using our E3SM-Hooks. In this case, the hook can be bypassed after ensuring that your commit message looks correct by issuing the following command:

git commit --no-verify


Integrating machine file changes

If the changes are only to machine files or the machine-specific blocks in config_machines.xml, config_compiler.xml or config_batch.xml then the POC of the machine (see Configuration Management) can integrate them even if they are the author of the branch.