Speculative Long-term Development

Owned by Robert Jacob
Last updated: Oct 25, 2020
5 min read

Speculative Long-term Development is any development intended for E3SM AND its not clear when or if it will be officially added to a future version of the model.  For this discussion, "Officially Added" means it is on master, can be turned on using appropriate configuration options, and tests of that feature must pass for any other development (the test becomes part of e3sm_developer or e3sm_integration). Examples of speculative long-term development include E3SM developments for 2+ versions ahead of current version,  developments from other programs (SciDAC, ECP).

The E3SM Infrastructure group has some recommended procedures for doing speculative development.  Which one to use depends on the nature of your code development.

IF your feature touches many files and alters data types of a main model, you should do direct development in master (explanation of that is below)

IF your feature is isolated in a single subroutine and you are not changing datatypes, you can probably use long-lived branches.

If your feature is not speculative – it will definitely be in E3SM – but will take a long time, you should develop directly in master.

Development in E3SM master (recommended)

  1. Split the long-term development in to pieces that can be checked in to master as they are finished (ask the IG for help on this if you want).
  2. Merge the development pieces back to master with PRs as is done for regular E3SM development.
  3. When you have enough code to execute some of your feature on master, add a test or tests for it.  These tests won't be added to the main E3SM test suites until the feature is "officially added".  If you have more then one, they could be part of a separate test suite.  Your PR must still pass the standard E3SMs testing.
  4. Develop the next piece of your feature with a new branch from the head of master.



# If not done yet
git clone git@github.com:E3SM-Project/E3SM.git
For each incremental step
git fetch
git checkout master
git pull
# make new branch
git checkout -b username/branchname
# ... develop
# ... test locally
# push changes to master
git add <newfiles>
git commit -a
git push -u origin username/branchname

# Now make PR from E3SM github page https://github.com/E3SM-Project/E3SM


Key points:  Feature is always in the E3SM source but may not be fully working.  Feature developer is always working with recent master to avoid divergence.   Least painful integration.   Developers of other features can build off of what you've committed to master.

Long-lived integration branch (for 2 or more related features)

  1. Create an integration branch which will track E3SM master.   This can be master on a fork of E3SM such as https://github.com/E3SM-Project/ACME-ECP
  2. Designate a person to maintain the integration branch.  They will update it with latest E3SM master.
  3. Make feature branches off of this integration branch.  PR's should be made back to the integration branch.
  4. Before merging to the integration branch, test each PR individually using e3sm_developer plus additional tests for the feature before merging.
  5. If testing each PR each day costs to much, you should also make a "next" branch for the integration branch to test multiple PRs nightly.  This may also be necessary if you want more assurance the integration branch is stable.
  6. The maintainer should make "mini releases" of the integration branch with a PR back to E3SM-Project/E3SM.  (tag the integration branch to mark these).

Key points:  Feature lives on the integration branch until integration branch is merged to upstream master.  Maintaining the integration branch (with merges from upstream master) could become arbitrarily hard if related codes diverge.  Allows team of developers to work on a large component and stay in sync with each other.  Rest of E3SM is unaware of developments until integration branch is merged to master.


Create fork from https://github.com/E3SM-Project/E3SM (use 'fork' button)

Development

# Let's assume new fork is https://github.com/MyProject/E3SM
git clone git@github.com/MyProject/E3SM.git
git fetch
git checkout master
git pull
# make new branch
git checkout -b username/branchname
# ... develop
# ... test locally
# push changes to master
git add <newfiles>
git commit -a
git push -u origin username/branchname

Syncing with E3SM master

reference: https://help.github.com/articles/syncing-a-fork/
$ git clone git@github.com/MyProject/E3SM.git
$ cd E3SM
$ git remote add upstream git@github.com:E3SM-Project/E3SM.git
# When updating fork from E3SM-Project/E3SM
$ git fetch upstream  (update local E3SM-Project/E3SM master, locally labelled 'upstream/master')
$ git checkout master (local MyProject/E3SM master)
$ git merge upstream/master (merge in changes from E3SM-Project/E3SM)
(resolve any conflicts)
$ git push origin master (push the merged changes to githup repo  git@github.com/MyProject/E3SM_Fork.git)

Long-lived branch off of E3SM-Project/E3SM master (or from an integration branch)

  1. Start a branch for your development as is done for any E3SM development.
  2. As you develop code on your branch, make a test or tests for your feature
  3. Keep your branch up-to-date by the following:
    1. PREFERRED: Periodically rebase your branch to the head of master (or head of the integration branch)
    2. OR you can merge master to your long-lived branch IF AND ONLY IF:
      1. A subroutine or function your feature depends on has changed its API OR
      2. A tag has been made on E3SM master (so you get a known state).   Do NOT make random "oh its been a while" merges of master to your branch.
      3. DO NOT merge from master just for machine config updates.  Cherry pick those or copy them.
  4. Run e3sm_developer on your branch along with tests for your feature as development proceeds.
  5. When finished, make a PR as for normal E3SM development.

Key points:  Features lives on your branch until it is finished.   Maintaining the branch (with rebases or occasional merge from master) could become arbitrarily hard if related codes diverge.  Other developers unaware of feature until its finished. 

3a PREFERRED: Periodically rebase your branch to the head of master (or head of the integration branch):

# If not done yet
git clone git@github.com:E3SM-Project/E3SM.git


# make new branch
git checkout -b username/branchname
(develop on username/branchname, probably with subbranches)
git push -u origin username/branchname
# Periodically update master and rebase work
git checkout master
git pull
git checkout username/branchname
git rebase master
(resolve any conflicts)
# push the new  rebased branch back to E3SM
git push origin username/branchname
(Tell everyone working on the branch that it's been rebased, everyone should now rename or
delete their local username/branch and check out a fresh version of username/branchname)


3b OR you can merge master to your long-lived branch (READ CONDITIONS ABOVE)


# If not done yet
git clone git@github.com:E3SM-Project/E3SM.git

# make new branch
git checkout -b username/branchname
(develop on username/branchname, probably with subbranches)
git push -u origin username/branchname

# when merging in master (READ CONDITIONS!)
git checkout username/branchname
git fetch
git merge master
git push