Development Big Picture

Owned by Robert Jacob
Last updated: May 03, 2024
3 min read

The Big Picture of E3SM Model Development (for E3SM staff)

All E3SM developers should be familiar with the basics of development with git including cloning the E3SM repo, making a branch, committing changes and submitting a Pull Request on Github.  Also writing good commit messages and pull request descriptions.  Read the Development Getting Started Guide and Commit and PR message template. For more info see, the Git Tutorial and Development Reference.

Group Leads and Epic leads coordinate and schedule development relevant to their group and/or epic.   They should have developers assigned to work on features in the order necessary to meet deliverables in the Roadmaps (the leads might also be the developers).  Epic leads should make sure developers are finishing their work and getting it merged to the E3SM "master" branch.

Mailing Lists:  Note that getting a confluence account does not automatically add you to the relevant mailing lists.   All users should periodically review the list of mailing lists and join any that may be relevant:  /wiki/spaces/ED/pages/818381294

Integrators:  every group has 2 or 3 Integrators who take completed feature branches and, if they meet the requirements, merge them to master.  See Integrator Guide for more info.

The Coupled Model Group establishes the timeline for integrating new features for the new release, relevant testing criteria, and when the master should be tagged (using the Branch, Tag, and Version name conventions) all timed to meet E3SM deliverables as defined in the Roadmaps.  Integrators should be aware of the Roadmap for their component.

A branch for every feature:  Group and task leads and developers should define units of code development that are big enough to be relevant to others, but simple enough that it can be called one feature (or perhaps a couple of highly-interdependent features) and make a branch for each unit of work.  These branches are called "feature branches" in our documentation.  The developer should make commits that are minimal atomic units of work to complete the feature that the branch seeks to implement.  This separation makes it easier to review and debug later.   New development should generally start as a branch from "master".  Avoid creating large branches that contain many features and touch dozens of files. If a large branch is needed for some major development, the Infrastructure Group should be consulted in planning.

Bug fix branches:  When a bug is found on master, the branch for developing a fix should start from the commit that introduced the bug.  This is so that any feature branch which also contains the bug can easily merge in the bug fix without having to merge in many other changes that may have occurred while the feature branch was being developed.

Focus on your branch:  While working on their branch, developers should focus on the work necessary to complete the branch and not worry about what else is going on.  In particular, there is almost never a need in git to "merge from master" or "stay current" with development going on in master or other branches to make the eventual merge easier.   If a developer thinks a merge from master or another branch is necessary to complete their work, they should clearly indicate why the need the merge in their commit message.

Developing with externals:  some code in E3SM has its primary development in another repo.  See Externals in the E3SM code

"master" is always stable:  Our code development workflow ensure that master is always stable and can be used to start new development.

Finishing development by Testing and Merging: When the feature branch is done, the developer makes sure the e3sm_developer test suite passes (or the failures are expected).  When the test results are in order, the developer issues a "pull request (PR)", and designates an Integrator (the Assignee in github).  The Integrator conducts the code review, merges the feature branch into the "next" integration branch and tests the feature within the latest version of E3SM using the e3sm_integration suite.  The integrator and developer may need to work together to resolve any issues that arise from the code review or testing in the "next" branch.  The integrator will then merge the branch into master.   See the Development Reference for more information.  In summary:

  • One feature per git branch (ideally)
  • One pull request (PR) per branch (absolutely)
  • Nothing gets added to "next" or "master" except by merging a branch through a PR (absolutely)

Coding with performance in mind: Fortran Performance Best Practices