Why subtree for CIME?
Very early in ACME, we decided that we want developers to get a working version of the code with a single "git clone" command. For that reason, we use subtree to maintain CIME within ACME because CIME must be present for building/running the model. Submodule requires an additional command after the git clone.
Terms
"the fork" refers to ACME's fork of CIME: https://github.com/ACME-Climate/cime (no longer used)
"the upstream repo" refers to ESMCI CIME: https://github.com/ESMCI/CIME
"the subtree" refers to all the code under https://github.com/ACME-Climate/ACME/cime
Initial add of CIME2 to ACME:
This was done once.
ACME was freshly cloned (from Sept 3, 2015 head: https://github.com/ACME-Climate/ACME/commit/c2b9b202f95cba910b72a53acc4c747beac316a7 and a remote for the ACME CIME fork was added:
git remote add -f --tags acmecime git@github.com:ACME-Climate/cime.git
CIME was added with this command:git subtree add --squash --prefix=cime/ acme-merge2.1
Subsequent commits did the following:
- move model components to their cime locations
- add cime build capability
- remove the models directory
- add ability to build component models with cime.
Summary of merges and splits involving CIME5
| direction | commit on master and date merged. PR link. | feature branch that was merged to master | how feature branch was created | notes |
|---|---|---|---|---|
| merge ESMCI to ACME | in ACME: /rljacob/cime/cime5-upgrade | in ACME: First removed cime2 in 8ed674f84ce Then did a subtree merge squash of 0eb331d in 7d7641505cc That squash was merge-committed to the branch in 15abd5853 | brought in CIME5.1.4 Did a bunch more commits on that branch to get everything working. | |
| split ACME to ESMCI | git subtree split c50a810d95442^.. --onto=0eb331d371 --prefix=cime/ -b rljacob/split-cime51-5 | send ACME's CIME5.1 changes to ESMCI | ||
| merge ESMCI to ACME | in ACME rljacob/cime/uptocime5.2.0 | brought in CIME5.2.0 215e459 is commit in ESMCI last brought to ACME, which is the 5.2.0 tag | ||
| split ACME to ESMCI | 2 on ESMCI: agsalin/cime52-with-acmesplit-03292017 acme commits merged to CIME 5.2 agsalin/merge-from-acme-03292017 Commits merged with ESMCI master | Starting from ACME master, commit f255704 by Az on March 28.
| Send ACME's CIME5.2 changes to ESMCI (Acme changes as of 2017-03-29 since last split) Notes: 628a26d is first commit in ACME after last merge Branch agsalin/split-03292017 not pushed (acme code sizeof cime repo?) | |
| merge ESMCI back to ACME | On ACME: agsalin/update-to-cime5.3 | git checkout -b agsalin/update-to-cime5.3 git fetch esmcicime git subtree pull --prefix=cime esmcicime master | Brought in CIME 5.3.0-alpha06 CIME master hash: 6156e0a tag: cime5.3.0-alpha.06-toacme01 To facilitate merge, I first moved all ACME files to the new cime dir name, before subtree pull. Brought in entire CIME commit history. | |
merge ESMCI to ACME | On ACME: agsalin/update-to-cime5.3.1 started 5/2/2107 | same as above | Brought in CIME 5.3.0-alpha.10 CIME master hash: 15297cd tag: cime5.3.0-alpha.10-toacme01 merge more ESMCI changes to ACME, without doing ACME→ESMCI first. | |
| split ACME to ESMCI | ACME commit: acme-split-06-01-2017 onto ESMCI commit: cime5.3.0-alpha.10-toacme01 | Change instructions to use tags | ||
| merge ESMCI to ACME | in ACME 2a095e62 11 Oct 2017 | On ACME: jgfouca/update_to_cime_5.3.0.34 | same as in previous ESMCI→ACME merges | brought in CIME 5.3.0.alpha34 |
| split ACME to ESMCI | Make script |
Add of CIME5 to ACME:
The existing cime was removed with git rm and the above repeated with
git remote add esmcicime git@github.com:ESMCI/cime.git
When modifications from within the subtree need be moved back to the upstream repo:
(The example code below is from the first time this was done)
NOTE: if your work on a branch requires changing files in a subtree, be sure to commit those changes separately. This will aid the future subtree split of those files back to the external.
First step is to create a branch that contains the changes you want to move back to the external (ESMCI/CIME in this case). The branchname should follow ACME conventions. NOTE: the command below must be issues from the top directory of ACME.
%%First Time Only: git subtree split -P cime/ -b rljacob/acmemerge-to-cime1 (first time) blogin1: git subtree split --prefix cime/ --ignore-joins -b rljacob/acmemerge-to-cime1 (after there's been merges back-and-forth). Created branch 'rljacob/acmemerge-to-cime1' fe8c483486af5b95f24ee64ce45ccb6d6fb6061e blogin1: git branch master rljacob/acmemerge-to-cime1 * rljacob/cime-merge
Make sure you have a name for the external's git repo. In below, "origin" is the usual ACME repo and "esmcicime" is the upstream repo of cime.
blogin1[102]: git remote add esmcicime git@github.com:ESMCI/CIME.git blogin1[102]: git remote -v esmcicime git@github.com:ESMCI/CIME.git (fetch) esmcicime git@github.com:ESMCI/CIME.git (push) origin git@github.com:ACME-Climate/ACME.git (fetch) origin git@github.com:ACME-Climate/ACME.git (push)
Now checkout the master of esmcicime. Note that the presence of the mpas-o module will cause a message.
blogin1[156]: git fetch esmcicime blogin1[156]: git checkout esmcicime/master warning: unable to rmdir components/mpas-o/model: Directory not empty Checking out files: 100% (7985/7985), done. Note: checking out 'esmcicime/master'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example: git checkout -b <new-branch-name> HEAD is now at b496e53... Merge branch 'jgfouca/skybridge_working' (PR #7) blogin1: git branch * (HEAD detached at esmcicime/master) master rljacob/acmemerge-to-cime1 rljacob/cime-merge
At this point, you're in a "Detached head state". You can ignore the message about mpas-o. mpas-o becomes an untracked file and will be ignored by git in the rest of the operations.
From the detached head state, make a branch from which you'll bring the new code to cime's master.
blogin1: git checkout -b rljacob/merge-from-acme1 Switched to a new branch 'rljacob/merge-from-acme1' blogin1]: git branch master rljacob/acmemerge-to-cime1 rljacob/cime-merge * rljacob/merge-from-acme1 blogin1[173]: git status On branch rljacob/merge-from-acme1 Untracked files: (use "git add <file>..." to include in what will be committed) components/mpas-o/ nothing added to commit but untracked files present (use "git add" to track) blogin1[174]:
Now merge onto the branch you just created the changes that are on the branch created by the subtree split (the first command above).
blogin1[175]: git merge rljacob/acmemerge_to_cime1
If you're lucky, there will be no conflicts and you'll be asked to make a commit message. If there are conflicts, resolve them with the usual procedure, add the resolved files, and commit (and finally enter the merge commit message).
You can now push the branch containing the code for the external and merge it to master using the external's procedures.
Two-step workflow option (under development as of March 2017)
Here are the commands for a 2-step process that Andy used successfully, March 2017. Need to integrate with above description once we settle on a process.
Previously, ESMCI/CIME was tagged with "cime5.2.0", and that code was brought into ACME. The process below (recommended by Rob) is a 2-step process. First the changes in ACME are merged into a branch coming off of that tag. A second step merges those changes into master. One nice part of this process is that the first step has mostly ACME changes, so conflicts are can mostly be reconciled by taking ACME version. The second step has just true conflicts, which will not be very many.
NOTE: need to pick better branch naming conventions in the future.
This setting can be needed.
$ git config merge.renameLimit 999999
get ACME code
$ git clone git@github.com:ACME-Climate/ACME.git
$ cd ACME
Add ESMCI/CIME as an additional remote.
$ git remote add esmcicime git@github.com:ESMCI/CIME.git
$ git fetch esmcicime --tags
Make a tag to mark split-point, it can be deleted once there's a new split point
$ git tag e3sm2cime-split-MM-DD-YYYY
$ git push origin e3sm2cime-split-MM-DD-YYYY
Create cime subtree of ACME. You will still be in a directory
called ACME, but it will now have the cime dir contents
NOTE: mm-dd-yyyy is the date of the PREVIOUS split
CIME5.2.0 is tag of last pull from ESMCI
$ git subtree split acme-split-mm-dd-yyyy.. --prefix=cime --onto=CIME5.2.0 --ignore-joins -b $USER/split-MMDDYYYY
Create a branch off of the CIME tag that ACME last incorporated,
and merge the acme/cime subtree changes onto that branch.
NOTE: YOU NEED TO PICK THE CORRECT TAG HERE
$ git checkout -b $USER/cime52-with-acmesplit-MMDDYYYY CIME5.2.0
$ git merge -X rename-threshold=25 $USER/split-MMDDYYYY
Resolve conflicts and commit (acme/cime changes into branch off of esmci/cime tag)
Most conflicts can be settled by taking ACME version, since we backed up CIME to the tag.
Git checkout --theirs <file> will take ACME version
$ <Numerous conflict edits, followed by "git add <file>" for each <file> with conflict>
$ git commit .
$ git push esmcicime agsalin/cime52-with-acmesplit-MMDDYYYY
Now, you have a branch of cime, off of the tag, with ACME changes added in
Create a branch off of CIME master, and merge the acme changes into it
$ git checkout esmcicime/master
$ git checkout -b agsalin/merge-from-acme-03292017
$ git merge agsalin/cime52-with-acmesplit-03292017
Resolve conflicts and commit (acme/cime changes added to CIME tag, merged into cime master)
$ <Numerous conflict edits, followed by "git add <file>" for each <file> with conflict>
$ ./scripts/tests/scripts_regression_tests.py
$ git push esmcicime agsalin/merge-from-acme-03292017
Issue PR
Delete old tag:
$ git tag -d acme-split-mm-dd-yyyy
$ git push origin :refs/tags/acme-split-mm-dd-yyyy