2022/4/8: Original
2022/6/17: Revised to stress importance of not bundling features in a single PR

...

• Coupled model: B-case runs (~100 years) for WC, cryosphere and BGC configurations

• Component models: F, I and G cases, typically much shorter (e.g. 5 year F case with cyclic year 2010 forcing)

• Updated periodically when climate changes are integrated, or monthly ( to check for unintended changes)

• New features will be evaluated with respect to these reference solutions, with documentation on the import metrics

Note: E3SM Reference solutions and related documentation not yet ready!

Section 1: New Feature Request Process and Documentation Requirements

...

1. Before writing code: Complete the New Feature Overview Document. Include, as appropriate:

   1. High-level description of code changes and/or new code, an overview of the design, infrastructure changes

   2. Expected improvements and how these will be demonstrated

   3. Describe needed updates to E3SM documentation

   4. Expected impacts on computational performance and mass/energy budgets.

   5. If relevant: describe papers that will be published

2. Before writing code: Document Review:

   1. Reviewed by component lead or their delegate. Reviewer should:

      1. Review document for completeness

      2. Determine if there is sufficient benefit to E3SM to justify the E3SM integration and future maintenance costs.

         1. For features not needed by the E3SM SFA, but which may be needed to support other DOE BER missions, the component lead should consult with E3SM leadership.

      3. Determine if review by performance and infrastructure groups if needed and ensure they are done.

         1. discourage and/or flag use of advanced language features or unnecessary complexity that may not be supported well by compilers or may cause performance degradation

      4. Create one confluence page per document to allow for discussion during the review process

         1. Confluence space for all new feature overview documents and their approval status.

3. After new feature is approved, begin work in an E3SM fork, following Development Getting Started Guide

   1. Features which are not approved but are needed for other reasons can be maintained by the developer on their E3SM fork.

4. When work is completed:

   1. Revise overview document to show improvements as originally proposed in overview document, and document simulation results from new feature PR guidelines. Include links to as yet to be constructed results archiving system

   2. Submit PR with links to documentation, follow new feature PR process.

...

PR’s are divided into 4 categories which require different levels of documentation and simulation results before a PR can be submitted.

1. BFB : Bit(bit-for-bit (): BFB ) changes which do not include stealth features.

2. Roundoff: These are usually infrastructure and code cleanup changes, compiler changes and system updates which introduce changes for which strong evidence can be provided that the changes do not effect the simulated climate.

3. Stealth Features: These are new features that are turned off by default and BFB with the previous code when turned off.

   1. Stealth features that are not BFB when turned off should be split into two PRs.

4. Climate Changing: PRs that result in larger than roundoff changes. These include new features turned on by default and also changes to default configurations which turn on stealth features.

...

1. New and stealth features: dont submit PR until Section1 approval of overview document

2. Submit the PR to the E3SM github site following Development Getting Started Guide . Links to the material outlined in Section 2 above should be in the first comment after the PR description.

3. For code with potential performance impacts: Performance group lead or delegate reviews supplied PACE data.

4. It is important not to bundle multiple features or fixes into a single PR.

5. Climate Changing PRs (new features and bug fixes)

   1. Simulation group lead approves based on climate changes

   2. Simulation group lead (or delegate) will be assigned as a reviewer as well as appropriate component lead and then verifies:

      1. source code review was completed

      2. component lead also approves

      3. PR includes test (or is covered by existing tests)

      4. climate changing evaluation process (from Section 2) completed and results archived

      5. Updates main branch reference B case as needed.

      6. PR contains a single feature ( to ensure multiple features are tested independently)

6. Stealth features

   1. Component lead (or delegate) is assigned as a reviewer and verifies:

      1. source code review

      2. PR includes appropriate test

      3. stealth-feature evaluation process (from Section 2) completed and results archived

      4. Expanded BFB tests were run with feature turned on

      5. PR contains a single feature ( to ensure multiple features are tested independently)

7. Roundoff level PRs

   1. Component lead (or delegate) verifies:

      1. source code review was completed

      2. evidence to support roundoff level tests is sufficient, if not, follow component new feature climate evaluation procedures

      3. no stealth features

8. BFB

   1. component lead (or delegate) be assigned as a reviewer verifies:

      1. source code review was completed

      2. no stealth features

...