Why subtree for CIME?
Very early in ACME, we decided that we want developers to get a working version of the code with a single "git clone" command. For that reason, we use subtree to maintain CIME within ACME because CIME must be present for building/running the model. Submodule requires an additional command after the git clone.
Terms
"the fork" refers to ACME's fork of CIME: https://github.com/ACME-Climate/cime (no longer used)
"the upstream repo" refers to ESMCI CIME: https://github.com/ESMCI/CIME
"the subtree" refers to all the code under https://github.com/ACME-Climate/ACME/cime
Initial add of CIME2 to ACME:
This was done once.
ACME was freshly cloned (from Sept 3, 2015 head: https://github.com/ACME-Climate/ACME/commit/c2b9b202f95cba910b72a53acc4c747beac316a7 and a remote for the ACME CIME fork was added:
git remote add -f --tags acmecime git@github.com:ACME-Climate/cime.git
CIME was added with this command:
git subtree add --squash --prefix=cime/ acme-merge2.1
Subsequent commits did the following:
- move model components to their cime locations
- add cime build capability
- remove the models directory
- add ability to build component models with cime.
Summary of merges and splits involving CIME5
direction | commit | feature branch | start of feature branch | notes |
---|---|---|---|---|
merge ESMCI to ACME | in ACME c50a810d9544251d50
| in ACME: /rljacob/cime/cime5-upgrade | in ACME: First removed cime2 in 8ed674f84ce Then did a subtree merge squash of 0eb331d in 7d7641505cc That squash was merge-committed to the branch in 15abd5853 | brought in CIME5.1.4 Did a bunch more commits on that branch to get everything working. |
split ACME to ESMCI | send ACME's CIME5.1 changes to ESMCI | |||
merge ESMCI to ACME | brought in CIME5.2.0 | |||
split ACME to ESMCI | Starting from ACME master, commit f255704 by Az on March 28. Notes: 628a26d is first commit in ACME after last split 215e459 is commit in ESMCI last brought to ACME, which is the 5.2.0 tag Branch agsalin/split-03292017 not pushed (acme code sizeof cime repo?) | On ESMCI: agsalin/cime52-with-acmesplit-03292017 acme commits merged to CIME 5.2 | On ESMCI: agsalin/merge-from-acme-03292017 Commits merged with master | Send ACME's CIME5.2 changes to ESMCI (Acme changes as of 03292017 since last split) |
merge ESMCI back to ACME | Starting in ACME master: git checkout -b agsalin/update-to-cime5.3 git fetch esmcicime git subtree pull --prefix=cime esmcicime master | On ACME: agsalin/update-to-cime5.3 | CIME master hash: 6156e0a tag: cime5.3.0-alpha.06-toacme01 | To facilitate merge, I first moved all ACME files to the new cime dir name, before subtree pull. |
merge more ESMCI changes to ACME, without doing ACME→ESMCI first. | same as above | On ACME: agsalin/update-to-cime5.3.1 | CIME master hash: ae2a5ca tag: TBD |
Add of CIME5 to ACME:
The existing cime was removed with git rm and the above repeated with
git remote add esmcicime git@github.com:ESMCI/cime.git
When modifications from within the subtree need be moved back to the upstream repo:
(The example code below is from the first time this was done)
NOTE: if your work on a branch requires changing files in a subtree, be sure to commit those changes separately. This will aid the future subtree split of those files back to the external.
First step is to create a branch that contains the changes you want to move back to the external (ESMCI/CIME in this case). The branchname should follow ACME conventions. NOTE: the command below must be issues from the top directory of ACME.
%%First Time Only: git subtree split -P cime/ -b rljacob/acmemerge-to-cime1 (first time) blogin1: git subtree split --prefix cime/ --ignore-joins -b rljacob/acmemerge-to-cime1 (after there's been merges back-and-forth). Created branch 'rljacob/acmemerge-to-cime1' fe8c483486af5b95f24ee64ce45ccb6d6fb6061e blogin1: git branch master rljacob/acmemerge-to-cime1 * rljacob/cime-merge
Make sure you have a name for the external's git repo. In below, "origin" is the usual ACME repo and "esmcicime" is the upstream repo of cime.
blogin1[102]: git remote add esmcicime git@github.com:ESMCI/CIME.git blogin1[102]: git remote -v esmcicime git@github.com:ESMCI/CIME.git (fetch) esmcicime git@github.com:ESMCI/CIME.git (push) origin git@github.com:ACME-Climate/ACME.git (fetch) origin git@github.com:ACME-Climate/ACME.git (push)
Now checkout the master of esmcicime. Note that the presence of the mpas-o module will cause a message.
blogin1[156]: git fetch esmcicime blogin1[156]: git checkout esmcicime/master warning: unable to rmdir components/mpas-o/model: Directory not empty Checking out files: 100% (7985/7985), done. Note: checking out 'esmcicime/master'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example: git checkout -b <new-branch-name> HEAD is now at b496e53... Merge branch 'jgfouca/skybridge_working' (PR #7) blogin1: git branch * (HEAD detached at esmcicime/master) master rljacob/acmemerge-to-cime1 rljacob/cime-merge
At this point, you're in a "Detached head state". You can ignore the message about mpas-o. mpas-o becomes an untracked file and will be ignored by git in the rest of the operations.
From the detached head state, make a branch from which you'll bring the new code to cime's master.
blogin1: git checkout -b rljacob/merge-from-acme1 Switched to a new branch 'rljacob/merge-from-acme1' blogin1]: git branch master rljacob/acmemerge-to-cime1 rljacob/cime-merge * rljacob/merge-from-acme1 blogin1[173]: git status On branch rljacob/merge-from-acme1 Untracked files: (use "git add <file>..." to include in what will be committed) components/mpas-o/ nothing added to commit but untracked files present (use "git add" to track) blogin1[174]:
Now merge onto the branch you just created the changes that are on the branch created by the subtree split (the first command above).
blogin1[175]: git merge rljacob/acmemerge_to_cime1
If you're lucky, there will be no conflicts and you'll be asked to make a commit message. If there are conflicts, resolve them with the usual procedure, add the resolved files, and commit (and finally enter the merge commit message).
You can now push the branch containing the code for the external and merge it to master using the external's procedures.
Two-step workflow option (under development as of March 2017)
Here are the commands for a 2-step process that Andy used successfully, March 2017. Need to integrate with above description once we settle on a process.
Previously, ESMCI/CIME was tagged with "cime5.2.0", and that code was brought into ACME. The process below (recommended by Rob) is a 2-step process. First the changes in ACME are merged into a branch coming off of that tag. A second step merges those changes into master. One nice part of this process is that the first step has mostly ACME changes, so conflicts are can mostly be reconciled by taking ACME version. The second step has just true conflicts, which will not be very many.
NOTE: need to pick better branch naming conventions in the future.
This setting can be needed. $ git config merge.renameLimit 999999 get ACME code $ git clone git@github.com:ACME-Climate/ACME.git $ cd ACME Add ESMCI/CIME as an additional remote. $ git remote add esmcicime git@github.com:ESMCI/CIME.git $ git fetch esmcicime Create cime subtree of ACME. You will still be in a directory called ACME, but it will now have the cime dir contents NOTE: 628a26d is first hast after last split from ACME 215e459 is hash of last pull from ESMCI, tag CIME5.2.0 $ git subtree split 628a26d^.. --prefix=cime --onto=215e459 --ignore-joins -b agsalin/split-03292017 Create a branch off of the CIME tag that ACME last incorporated, and merge the acme/cime subtree changes onto that branch. NOTE: YOU NEED TO PICK THE CORRECT TAG HERE $ git checkout tags/cime5.2.0 $ git checkout -b agsalin/cime52-with-acmesplit-03292017 $ git merge -X rename-threshold=25 agsalin/split-03292017 Resolve conflicts and commit (acme/cime changes into branch off of esmci/cime tag) Most conflicts can be settled by taking ACME version, since we backed up CIME to the tag. Git checkout --theirs <file> will take ACME version $ <Numerous conflict edits, followed by "git add <file>" for each <file> with conflict> $ git commit . $ git push esmcicime agsalin/cime52-with-acmesplit-03292017 Now, you have a branch of cime, off of the tag, with ACME changes added in Create a branch off of CIME master, and merge the acme changes into it $ git checkout esmcicime/master $ git checkout -b agsalin/merge-from-acme-03292017 $ git merge agsalin/cime52-with-acmesplit-03292017 Resolve conflicts and commit (acme/cime changes added to CIME tag, merged into cime master) $ <Numerous conflict edits, followed by "git add <file>" for each <file> with conflict> $ ./scripts/tests/scripts_regression_tests.py $ git push esmcicime agsalin/merge-from-acme-03292017 Issue PR