{"type":"doc","content":[{"type":"paragraph","content":[{"text":"Contact: ","type":"text"},{"type":"mention","attrs":{"id":"5d8124e3d7f1360c34f0ce6e","text":"@Ryan Forsyth"}}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Summary","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Useful Aliases","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The aliases provided in this section may be useful.","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Configuring the Model Run – Run Script","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The run-script remains mostly the same between runs. Users can change configuration parameters in it though.","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Running the Model","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"It is recommended to run a few short tests before starting a production simulation.","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Short Term Archiving","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"To avoid having so many output files in one directory, we can archive output into appropriate subdirectories.","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Performance Information","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"PACE is used to display performance information from a run.","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Re-Submitting a Job After a Crash","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"If a job crashes, it is easy to re-submit after making a fix.","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Post-Processing with ","type":"text"},{"text":"zppy","type":"text","marks":[{"type":"code"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"zppy","type":"text","marks":[{"type":"code"}]},{"text":" is a tool that brings multiple post-processing tools together – with ","type":"text"},{"text":"zppy","type":"text","marks":[{"type":"code"}]},{"text":", users can easily run E3SM Diags and MPAS-Analysis as well as generate global time series plots.","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Documenting the Model Run","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Model runs should be documented on its own Confluence page providing run details and results.","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Long Term Archiving with ","type":"text"},{"text":"zstash","type":"text","marks":[{"type":"code"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"zstash","type":"text","marks":[{"type":"code"}]},{"text":" can be used to archive results for the long-term.","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Publishing the simulation data (Optional)","type":"text"}]}]}]}]},{"type":"paragraph"},{"type":"extension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"toc","parameters":{"macroParams":{},"macroMetadata":{"macroId":{"value":"c3f7654b-217b-42cc-b615-a65266d1d8e3"},"schemaVersion":{"value":"1"},"title":"Table of Contents"}},"localId":"b7244a78-0fc2-46ab-a2c3-024764d22a9a"}},{"type":"heading","attrs":{"level":1},"content":[{"text":"Useful Aliases","type":"text"}]},{"type":"paragraph","content":[{"text":"Setting some aliases may be useful in running the model. You can edit ","type":"text"},{"text":"~/.bashrc","type":"text","marks":[{"type":"code"}]},{"text":" to add aliases. Run ","type":"text"},{"text":"source ~/.bashrc","type":"text","marks":[{"type":"code"}]},{"text":" to start using them.","type":"text"}]},{"type":"paragraph","content":[{"text":"Note that the specific file name may differ amongst machines. Other possibilities include ","type":"text"},{"text":"~/.bash_profile","type":"text","marks":[{"type":"code"}]},{"text":", ","type":"text"},{"text":"~/.bashrc.ext","type":"text","marks":[{"type":"code"}]},{"text":", or ","type":"text"},{"text":"~/.bash_profile.ext","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Batch jobs","type":"text"}]},{"type":"paragraph","content":[{"text":"To check on all batch jobs: ","type":"text"}]},{"type":"paragraph","content":[{"text":"alias sqa='squeue -o \"%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %.8Q %j\" --sort=P,-t,-p'","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"To check on your batch jobs:","type":"text"}]},{"type":"paragraph","content":[{"text":"alias sq='sqa -u $USER'","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"The output of ","type":"text"},{"text":"sq","type":"text","marks":[{"type":"code"}]},{"text":" uses several abbreviations: ST = Status, R = running, PD = pending, CG = completing.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Directories","type":"text"}]},{"type":"paragraph","content":[{"text":"You will be working in several directories. ","type":"text"}]},{"type":"paragraph","content":[{"text":"On ","type":"text"},{"text":"anvil","type":"text","marks":[{"type":"strong"}]},{"text":" or ","type":"text"},{"text":"chrysalis","type":"text","marks":[{"type":"strong"}]},{"text":" (i.e., LCRC machines), those paths are:","type":"text"}]},{"type":"paragraph","content":[{"text":": ","type":"text"},{"text":"${HOME}/E3SM/scripts","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":": ","type":"text"},{"text":"${HOME}/E3SM/code","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":": ","type":"text"},{"text":"/lcrc/group/e3sm//E3SMv2","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"So, it may be useful to set the following aliases:","type":"text"}]},{"type":"codeBlock","content":[{"text":"# Model running\nalias run_scripts=\"cd ${HOME}/E3SM/scripts/\"\nalias simulations=\"cd /lcrc/group/e3sm//E3SMv2/\"","type":"text"}]},{"type":"paragraph","content":[{"text":"On other machines, the paths are the same, except for the .","type":"text"}]},{"type":"paragraph","content":[{"text":"On ","type":"text"},{"text":"compy","type":"text","marks":[{"type":"strong"}]},{"text":" (PNNL):","type":"text"}]},{"type":"paragraph","content":[{"text":": ","type":"text"},{"text":"/compyfs//E3SMv2","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"On ","type":"text"},{"text":"cori","type":"text","marks":[{"type":"strong"}]},{"text":" (NERSC):","type":"text"}]},{"type":"paragraph","content":[{"text":": ","type":"text"},{"text":"${CSCRATCH}/E3SMv2","type":"text","marks":[{"type":"code"}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Configuring the Model Run – Run Script","type":"text"}]},{"type":"paragraph","content":[{"text":"Start with an example of a run script for a low-resolution coupled simulation:","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"02f88d7e-6a2d-4076-b44a-5ab924e0b967"}}]},{"text":" ","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The template script in the E3SM repository, which uses ","type":"text"},{"text":"cori-knl","type":"text","marks":[{"type":"code"}]},{"text":": ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://github.com/E3SM-Project/E3SM/blob/master/run_e3sm.template.sh"}},{"text":" ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"This guide uses a similar script, which uses ","type":"text"},{"text":"chrysalis","type":"text","marks":[{"type":"code"}]},{"text":": ","type":"text"},{"text":"https://github.com/E3SM-Project/SimulationScripts/blob/master/archive/v2/run.v2.LR.piControl.sh","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/E3SM-Project/SimulationScripts/blob/master/archive/v2/run.v2.LR.piControl.sh"}}]},{"text":" (Note: this is a private repo) ","type":"text"},{"text":".","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"1426e030-bf9c-4e77-9cd6-d2c6b33c1d36"}}]},{"text":" ","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Create a new run script or copy an existing one. The path to it should be ","type":"text"},{"text":"/run..sh","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"# Machine and project","type":"text","marks":[{"type":"code"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly MACHINE=chrysalis","type":"text","marks":[{"type":"code"}]},{"text":": the name of the machine you’re running on.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly PROJECT=\"e3sm\"","type":"text","marks":[{"type":"code"}]},{"text":": SLURM project accounting (typically e3sm).","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"# Simulation","type":"text","marks":[{"type":"code"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly COMPSET=\"WCYCL1850\"","type":"text","marks":[{"type":"code"}]},{"text":" : compset (configuration)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly RESOLUTION=\"","type":"text","marks":[{"type":"code"}]},{"text":"ne30pg2_EC30to60E2r2","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"c20181a4-d68f-419d-ae51-a4f4bab28534"}},{"type":"code"}]},{"text":"\"","type":"text","marks":[{"type":"code"}]},{"text":": resolution (low-resolution coupled simulation in this case)","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"ne30","type":"text","marks":[{"type":"code"}]},{"text":" is the number of spectral elements for the atmospheric dynamics grid, while ","type":"text"},{"text":"pg2","type":"text","marks":[{"type":"code"}]},{"text":" refers to the physics grid option. This mesh grid spacing is approximately 110 km.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"EC30to60E2r2","type":"text","marks":[{"type":"code"}]},{"text":" is the ocean and sea-ice resolution. The grid spacing varies between 30 and 60 km.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"For simulations with regionally refined meshes such as the N American atmosphere grid coupled to the WC14 ocean and sea-ice, replace with ","type":"text"},{"text":"northamericax4v1pg2_WC14to60E2r3","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly CASE_NAME=\"v2.LR.piControl\"","type":"text","marks":[{"type":"code"}]},{"text":": ","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"v2.LR","type":"text","marks":[{"type":"code"}]},{"text":" is a short custom description to help identify the simulation. ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"piControl","type":"text","marks":[{"type":"code"}]},{"text":" is the type of simulation. ","type":"text"},{"text":"Other options here include ,","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"1335db71-6caf-46e7-9f37-ab51705e4bcc"}}]},{"text":" but are not limited to: ","type":"text"},{"text":"amip","type":"text","marks":[{"type":"code"}]},{"text":", ","type":"text"},{"text":"F2010","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly CASE_GROUP=\"v2.LR\"","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"580b7cf9-2ea8-486e-8261-4b25ffc21493"}},{"type":"code"}]},{"text":":","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"580b7cf9-2ea8-486e-8261-4b25ffc21493"}}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"This will let you mark multiple cases as part of the same group for later processing (e.g., with PACE).","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Note: If this is part of a simulation campaign, ask your group lead about using a case_group label. Otherwise, please use a unique name to distinguish from existing case_group label names, i.g. “v2.LR“.","type":"text"}]}]}]}]}]},{"type":"paragraph","content":[{"text":"# Code and compilation","type":"text","marks":[{"type":"code"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly CHECKOUT=\"20210702\"","type":"text","marks":[{"type":"code"}]},{"text":": date the code was checked out on in the form ","type":"text"},{"text":"{year}{month}{day}","type":"text","marks":[{"type":"code"}]},{"text":". The source code will be checked out in sub-directory named ","type":"text"},{"text":"{year}{month}{day}","type":"text","marks":[{"type":"code"}]},{"text":" under .","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly BRANCH=\"master\"","type":"text","marks":[{"type":"code"}]},{"text":": branch the code was checked out from. ","type":"text"},{"text":"Valid options include “master”, a branch name, or a git hash.","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"55fca5af-95dd-4e6a-a391-da3afc492e66"}}]},{"text":" For provenance purposes, it is best to specify the ","type":"text"},{"text":"git hash","type":"text","marks":[{"type":"strong"}]},{"text":".","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly DEBUG_COMPILE=false","type":"text","marks":[{"type":"code"}]},{"text":" : option to compile with DEBUG flag (leave set to false)","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"# Run options","type":"text","marks":[{"type":"code"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly MODEL_START_TYPE=\"hybrid\"","type":"text","marks":[{"type":"code"}]},{"text":" : specify how the model should start – use ","type":"text"},{"text":"initial","type":"text","marks":[{"type":"code"}]},{"text":" conditions, ","type":"text"},{"text":"continue","type":"text","marks":[{"type":"code"}]},{"text":" from existing restart files, ","type":"text"},{"text":"branch","type":"text","marks":[{"type":"code"}]},{"text":", or ","type":"text"},{"text":"hybrid","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly START_DATE=\"0001-01-01\"","type":"text","marks":[{"type":"code"}]},{"text":" : model start date. Typically year 1 for simulations with perpetual (time invariant) forcing or a real year for simulation for transient forcings.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"# Set paths","type":"text","marks":[{"type":"code"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly CODE_ROOT=\"${HOME}/E3SMv2/code/${CHECKOUT}\"","type":"text","marks":[{"type":"code"}]},{"text":": where the E3SM code will be checked out.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly CASE_ROOT=\"/lcrc/group/e3sm/${USER}/E3SMv2/${CASE_NAME}\"","type":"text","marks":[{"type":"code"}]},{"text":": where the results will go. The directory ","type":"text"},{"text":"${CASE_NAME}","type":"text","marks":[{"type":"code"}]},{"text":" will be in ","type":"text"},{"text":"","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"# Sub-directories","type":"text","marks":[{"type":"code"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly CASE_BUILD_DIR=${CASE_ROOT}/build","type":"text","marks":[{"type":"code"}]},{"text":" : all the compilation files, including the executable.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly CASE_ARCHIVE_DIR=${CASE_ROOT}/archive","type":"text","marks":[{"type":"code"}]},{"text":" : where short-term archived files will reside.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"# Define type of run","type":"text","marks":[{"type":"code"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly run='production'","type":"text","marks":[{"type":"code"}]},{"text":": type of simulation to run – i.e, a short test for verification or a long production run. (See next section for details).","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"# Coupler history","type":"text","marks":[{"type":"code"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly HIST_OPTION=\"nyears\"","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly HIST_N=\"5\"","type":"text","marks":[{"type":"code"}]}]}]}]},{"type":"paragraph","content":[{"text":"# Leave empty (unless you understand what it does)","type":"text","marks":[{"type":"code"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly OLD_EXECUTABLE=\"\"","type":"text","marks":[{"type":"code"}]},{"text":" : this is a somewhat risky option that allows you to re-use a pre-existing executable. This is not recommended because it breaks provenance.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"# --- Toggle flags for what to do ----","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"This section controls what operations the script should perform. The run-e3sm script can be invoked multiple times with the user having the option to bypass certain steps by toggling ","type":"text"},{"text":"true","type":"text","marks":[{"type":"code"}]},{"text":" / ","type":"text"},{"text":"false","type":"text","marks":[{"type":"code"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"do_fetch_code=false","type":"text","marks":[{"type":"code"}]},{"text":" : fetch the source code from Github.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"do_create_newcase=true","type":"text","marks":[{"type":"code"}]},{"text":" : create new case.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"do_case_setup=true","type":"text","marks":[{"type":"code"}]},{"text":" : case setup.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"do_case_build=false","type":"text","marks":[{"type":"code"}]},{"text":" : compile.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"do_case_submit=true","type":"text","marks":[{"type":"code"}]},{"text":" : submit simulation.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"The first time the script is called, all the flags should be set to ","type":"text"},{"text":"true","type":"text","marks":[{"type":"code"}]},{"text":". Subsequently, the user may decide to bypass code checkout (","type":"text"},{"text":"do_fetch_code=false)","type":"text","marks":[{"type":"code"}]},{"text":" or compilation (","type":"text"},{"text":"do_case_build=false","type":"text","marks":[{"type":"code"}]},{"text":"). A user may also prefer to manually submit the job by setting ","type":"text"},{"text":"do_case_submit=false","type":"text","marks":[{"type":"code"}]},{"text":" and then invoking ","type":"text"},{"text":"./case.submit.","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph"},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"A case is tied to one code base and one executable. That is, if you change ","type":"text"},{"text":"CHECKOUT","type":"text","marks":[{"type":"code"}]},{"text":" or ","type":"text"},{"text":"BRANCH","type":"text","marks":[{"type":"code"}]},{"text":", then you should also change ","type":"text"},{"text":"CASE_NAME","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Running the Model","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Short tests","type":"text"}]},{"type":"paragraph","content":[{"text":"Before starting a long production run, it is ","type":"text"},{"text":"highly recommended","type":"text","marks":[{"type":"strong"}]},{"text":" to perform a few short tests to verify:","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The model starts without errors.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The model produces BFB (bit-for-bit) results after a restart.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The model produces BFB results when changing PE layout.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"(1) can spare you from a considerable amount of frustration. Imagine submitting a large job on a Friday afternoon, only to discover Monday morning that the job started to run on Friday evening and died within seconds because of a typo in a namelist variable or input file.","type":"text"}]},{"type":"paragraph","content":[{"text":"Many code bugs can be caught with (2) and (3). While the E3SM nightly tests should catch such non-BFB errors, it is possible that you’ll be running a slightly different configuration (for example a different physics option) for which those tests have not been performed.","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Running short tests","type":"text"}]},{"type":"paragraph","content":[{"text":"The type of run to perform is controlled by the script variable ","type":"text"},{"text":"run","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"You should typically perform at least two short tests (two different layouts, with and without restart).","type":"text"}]},{"type":"paragraph","content":[{"text":"First, let’s start with a short test using the 'S' (small) PE layout and running for 2x5 days:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly run='S_2x5_ndays'","type":"text","marks":[{"type":"code"}]}]}]}]},{"type":"paragraph","content":[{"text":"If you have not fetched and compiled the code, set all the toggle flags to true:","type":"text"}]},{"type":"codeBlock","content":[{"text":"do_fetch_code=true\ndo_create_newcase=true\ndo_case_setup=true\ndo_case_build=true\ndo_case_submit=true","type":"text"}]},{"type":"paragraph","content":[{"text":"At this point, execute the run-e3sm script:","type":"text"}]},{"type":"codeBlock","content":[{"text":"cd \n./run..sh","type":"text"}]},{"type":"paragraph","content":[{"text":"Fetching the code and compiling it will take some time (30 to 45 minutes), so go ahead a brew yourself a fresh cup of coffee. Once the script finished, the test job will have been submitted to the batch queue.","type":"text"}]},{"type":"paragraph","content":[{"text":"You can immediately edit the script to prepare for the second short test. In this case, we will be running for 10 days (without restart) using the 'M' (medium PE layout):","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly run='M_1x10_ndays'","type":"text","marks":[{"type":"code"}]}]}]}]},{"type":"paragraph","content":[{"text":"Since the code has already been fetched and compiled, change the toggle flags:","type":"text"}]},{"type":"codeBlock","content":[{"text":"do_fetch_code=false\ndo_create_newcase=true\ndo_case_setup=true\ndo_case_build=false\ndo_case_submit=true","type":"text"}]},{"type":"paragraph","content":[{"text":"and execute the script","type":"text"}]},{"type":"codeBlock","content":[{"text":"cd \n./run..sh","type":"text"}]},{"type":"paragraph","content":[{"text":"Since we are bypassing the code fetch and compilation (by re-using the previous executable), the script should only take a few seconds to run and again should submit the second test.","type":"text"}]},{"type":"paragraph","content":[{"text":"Note that short tests use separate output directories, so it is safe to submit and run multiple tests at once. If you’d like, you could submit additional test, for example 10 days with the medium 80 nodes ('M80') layout (","type":"text"},{"text":"M80_1x10_ndays","type":"text","marks":[{"type":"code"}]},{"text":").","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Verifying results are BFB","type":"text"}]},{"type":"paragraph","content":[{"text":"Once the short tests are complete, we can confirm the results were bit-for-bit (BFB) the same. All the test output is located under the ","type":"text"},{"text":"tests","type":"text","marks":[{"type":"code"}]},{"text":" sub-directory:","type":"text"}]},{"type":"codeBlock","content":[{"text":"cd //tests\nfor test in *\ndo\n zgrep -h '^ nstep, te ' ${test}/run/atm.log.*.gz | uniq > atm_${test}.txt\ndone\nmd5sum *.txt\n5bdfee6da8433cde08f33f3c046653c6 atm_M_1x10_ndays.txt\n5bdfee6da8433cde08f33f3c046653c6 atm_M80_1x10_ndays.txt\n5bdfee6da8433cde08f33f3c046653c6 atm_S_2x5_ndays.txt","type":"text"}]},{"type":"paragraph","content":[{"text":"To verify that the results are indeed BFB, ","type":"text"},{"text":"we extract global integral from the atmosphere log files (lines starting with ‘nstep, te’) and make sure that they are identical for all tests.","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"32100163-74f8-4ce9-83b2-1cb13e13276b"}}]}]},{"type":"paragraph","content":[{"text":"If the BFB check fails, you should stop here and understand why. If they succeed, you can now start the production simulation.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Production simulation","type":"text"}]},{"type":"paragraph","content":[{"text":"To prepare for the long production simulation, edit the run e3sm script and set:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly run='production'","type":"text","marks":[{"type":"code"}]}]}]}]},{"type":"paragraph","content":[{"text":"In addition, you may need to customize the code block below that some variables in the code block below to configure run options:","type":"text"}]},{"type":"paragraph","content":[{"text":"# Production simulation","type":"text","marks":[{"type":"code"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly PELAYOUT=\"M\"","type":"text","marks":[{"type":"code"}]},{"text":": ","type":"text"},{"text":"1=single processor, S=small, M=medium, L=large, X1=very large, X2=very very large","type":"text","marks":[{"type":"code"}]},{"text":". Production simulations typically use ","type":"text"},{"text":"M","type":"text","marks":[{"type":"code"}]},{"text":" or ","type":"text"},{"text":"L","type":"text","marks":[{"type":"code"}]},{"text":". The size determines how many nodes will be used. The exact number of nodes will differ amongst machines.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly WALLTIME=\"28:00:00\"","type":"text","marks":[{"type":"code"}]},{"text":" : maximum wall clock time requested for the batch jobs.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly STOP_OPTION=\"nyears\"","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly STOP_N=\"20\"","type":"text","marks":[{"type":"code"}]},{"text":" : units and length of each segment (i.e. each batch job)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly REST_OPTION=\"nyears\"","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly REST_N=\"5\"","type":"text","marks":[{"type":"code"}]},{"text":" : units and frequency for writing restart files (make sure ","type":"text"},{"text":"STOP_N","type":"text","marks":[{"type":"code"}]},{"text":" is a multiple of ","type":"text"},{"text":"REST_N","type":"text","marks":[{"type":"code"}]},{"text":", otherwise the model will stop without writing a restart fie at the end).","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly RESUBMIT=”9”","type":"text","marks":[{"type":"code"}]},{"text":" : number of resubmissions beyond the original segment. This simulation would run for a total of 200 years (20 + 9x20).","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"readonly DO_SHORT_TERM_ARCHIVING=false","type":"text","marks":[{"type":"code"}]},{"text":" : leave set to ","type":"text"},{"text":"false","type":"text","marks":[{"type":"code"}]},{"text":" if you want to manually run the short term archive.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Since the code has already been fetched and compiled for the short tests, the toggle flags can be set to:","type":"text"}]},{"type":"codeBlock","content":[{"text":"do_fetch_code=false\ndo_create_newcase=true\ndo_case_setup=true\ndo_case_build=false\ndo_case_submit=true","type":"text"}]},{"type":"paragraph","content":[{"text":"Finally, execute the script","type":"text"}]},{"type":"codeBlock","content":[{"text":"cd \n./run..sh","type":"text"}]},{"type":"paragraph","content":[{"text":"The script will automatically submit the first job. New jobs will be automatically be resubmitted at the end until the total number of segments have been run.","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Looking at Results","type":"text"}]},{"type":"codeBlock","content":[{"text":"cd /\nls","type":"text"}]},{"type":"paragraph","content":[{"text":"Explanation of directories:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"build","type":"text","marks":[{"type":"code"}]},{"text":": all the stuff to compile. The executable (","type":"text"},{"text":"e3sm.exe","type":"text","marks":[{"type":"code"}]},{"text":") is also there. ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"case_scripts","type":"text","marks":[{"type":"code"}]},{"text":": the files for your particular simulation. ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"run","type":"text","marks":[{"type":"code"}]},{"text":": where all the output will be. Most components (atmosphere, ocean, etc.) have their own log files. The coupler exchanges information between the components. The top level log file will be of the form ","type":"text"},{"text":"run/e3sm.log.*","type":"text","marks":[{"type":"code"}]},{"text":". Log prefixes correspond to components of the model:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"atm","type":"text","marks":[{"type":"code"}]},{"text":": atmosphere","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"cpl","type":"text","marks":[{"type":"code"}]},{"text":": coupler","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"ice","type":"text","marks":[{"type":"code"}]},{"text":": sea ice","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"lnd","type":"text","marks":[{"type":"code"}]},{"text":": land","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"ocn","type":"text","marks":[{"type":"code"}]},{"text":": ocean","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"rof","type":"text","marks":[{"type":"code"}]},{"text":": river runoff","type":"text"}]}]}]}]}]},{"type":"paragraph","content":[{"text":"Run ","type":"text"},{"text":"tail -f run/.log.","type":"text","marks":[{"type":"code"}]},{"text":" to keep up with a log in real time.","type":"text"}]},{"type":"paragraph","content":[{"text":"You can use the ","type":"text"},{"text":"sq","type":"text","marks":[{"type":"code"}]},{"text":" alias defined in the “Useful Aliases” section to check on the status of the job. The ","type":"text"},{"text":"NODE","type":"text","marks":[{"type":"code"}]},{"text":" in the output indicates the number of nodes used and is dependent on the ","type":"text"},{"text":"processor_config","type":"text","marks":[{"type":"code"}]},{"text":" / ","type":"text"},{"text":"PELAYOUT","type":"text","marks":[{"type":"code"}]},{"text":"size. ","type":"text"}]},{"type":"paragraph","content":[{"text":"When running on two different machines (such as Compy and Chrysalis) and/or two different compilers, the answers will not be the same, bit-for-bit. It is not possible using floating point operations to get bit-or-bit identical results across machines/compilers.","type":"text"}]},{"type":"paragraph","content":[{"text":"Logs being compressed to ","type":"text"},{"text":".gz","type":"text","marks":[{"type":"code"}]},{"text":" files is one of the last steps before the job is done and will indicate successful completion of the segment. ","type":"text"},{"text":"less .gz","type":"text","marks":[{"type":"code"}]},{"text":" will let you directly look at a gzipped log.","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Short Term Archiving","type":"text"}]},{"type":"paragraph","content":[{"text":"By default, E3SM will store all output files under the ","type":"text"},{"text":"run/","type":"text","marks":[{"type":"code"}]},{"text":" sub-directory. For long simulations, there could 10,000s to 100,000s of output files. Having so many files in a single directory can be very impractical, slowing down simple operations like ","type":"text"},{"text":"ls","type":"text","marks":[{"type":"code"}]},{"text":" to a crawl. CIME includes a short-term archiving utility that will neatly organize output files ","type":"text"},{"text":"into a separate ","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"9716870f-4035-40a7-940a-1575d957f0f6"}}]},{"text":"archive/","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"9716870f-4035-40a7-940a-1575d957f0f6"}},{"type":"code"}]},{"text":" sub-directory","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"9716870f-4035-40a7-940a-1575d957f0f6"}}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"Short term archiving can be accomplished with the following steps. This can be done while the model is still running.","type":"text"}]},{"type":"paragraph","content":[{"text":"Use ","type":"text"},{"text":"--force-move","type":"text","marks":[{"type":"code"}]},{"text":" to move instead of copying, which can take a long time. Set ","type":"text"},{"text":"--last-date","type":"text","marks":[{"type":"code"}]},{"text":" to the latest date in the simulation you want to archive. You do not have to specify a beginning date.","type":"text"}]},{"type":"codeBlock","content":[{"text":"cd //case_scripts\n./case.st_archive --last-date 0051-01-01 --force-move --no-incomplete-logs\ncd //archive\nls","type":"text"}]},{"type":"paragraph","content":[{"text":"Each component of the model has a subdirectory in ","type":"text"},{"text":"archive","type":"text","marks":[{"type":"code"}]},{"text":". There are also two additional subdirectories: ","type":"text"},{"text":"logs","type":"text","marks":[{"type":"code"}]},{"text":" holds the gzipped log files and ","type":"text"},{"text":"rest","type":"text","marks":[{"type":"code"}]},{"text":" holds the restart files.","type":"text"}]},{"type":"table","attrs":{"layout":"default","localId":"100911d4-72d4-494b-8ed0-f9bad50eae5a"},"content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"Component","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"Subdirectory","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"Files in the Subdirectory","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"31477a3b-d52d-4b47-9a25-4ebb5877ec92"}},{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"0782bf30-39ad-4859-b240-656a2cb63f1d"}},{"type":"strong"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"Atmosphere (Earth Atmospheric Model)","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"atm/hist","type":"text","marks":[{"type":"code"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"*.eam.h*","type":"text","marks":[{"type":"code"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"Coupler","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"cpl/hist","type":"text","marks":[{"type":"code"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"*.cpl.h*","type":"text","marks":[{"type":"code"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"Sea Ice (MPAS-Sea-Ice)","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"ice/hist","type":"text","marks":[{"type":"code"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"*.mpassi.hist.*","type":"text","marks":[{"type":"code"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"Land (Earth Land Model)","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"lnd/hist","type":"text","marks":[{"type":"code"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"*.elm.h*","type":"text","marks":[{"type":"code"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"Ocean (MPAS-Ocean)","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"ocn/hist","type":"text","marks":[{"type":"code"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"*.mpaso.hist.*","type":"text","marks":[{"type":"code"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"River Runoff (MOSART)","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"rof/hist","type":"text","marks":[{"type":"code"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"*.mosart.h*","type":"text","marks":[{"type":"code"}]}]}]}]}]},{"type":"paragraph"},{"type":"heading","attrs":{"level":1},"content":[{"text":"Performance Information","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"c49cecf5-09c9-4f51-8489-5f51a9a58cda"}}]}]},{"type":"paragraph","content":[{"text":"Model throughput is the number of simulated years per day. You can find this with:","type":"text"}]},{"type":"codeBlock","content":[{"text":"cd //case_scripts/timing\ngrep \"simulated_years\" e3sm*","type":"text"}]},{"type":"paragraph","content":[{"text":"PACE provides detailed performance information. Go to ","type":"text"},{"text":"https://pace.ornl.gov/ ","type":"text","marks":[{"type":"link","attrs":{"href":"https://pace.ornl.gov/ "}}]},{"text":"and enter your username to search for your jobs. You can also simply search by providing the JobID appended to log files (NNNNN.yymmdd-hhmmss where NNNNN is the Slurm job id). Click on a job ID to see its performance details. “Experiment Details” are listed at the top of the job’s page. There is also a helpful chart detailing how many processors and how much time each component (","type":"text"},{"text":"atm","type":"text","marks":[{"type":"code"}]},{"text":", ","type":"text"},{"text":"ocn","type":"text","marks":[{"type":"code"}]},{"text":", etc.) used. White areas indicate time spent idle/waiting. The area of each box is essentially the \"cost = simulation time * number of processors\" of the corresponding component.","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Re-Submitting a Job After a Crash","type":"text"}]},{"type":"paragraph","content":[{"text":"If a job crashes, you can rerun with:","type":"text"}]},{"type":"codeBlock","content":[{"text":"cd //case_scripts\n# Make any changes necessary to avoid the crash\n./case.submit","type":"text"}]},{"type":"paragraph","content":[{"text":"If you need to change a XML value, the following commands in the ","type":"text"},{"text":"case_scripts","type":"text","marks":[{"type":"code"}]},{"text":" directory are useful:","type":"text"}]},{"type":"codeBlock","content":[{"text":"> ./xmlquery # Get value of a variable\n> ./xmlchange -id -val # Set value of a variable","type":"text"}]},{"type":"paragraph","content":[{"text":"Before re-submitting:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Check that the rpointer files all point to the last restart. On very rare occasions, there might be some inconsistency if the model crashed at the end.","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Run ","type":"text"},{"text":"head -n 1 rpointer.*","type":"text","marks":[{"type":"code"}]},{"text":" to see the restart date.","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"gzip all the ","type":"text"},{"text":"*.log","type":"text","marks":[{"type":"code"}]},{"text":" files from the faulty segment so that they get moved during the next short-term archiving. To ","type":"text"},{"text":"gzip","type":"text","marks":[{"type":"code"}]},{"text":" log files from failed jobs, run ","type":"text"},{"text":"gzip *.log.*","type":"text","marks":[{"type":"code"}]},{"text":" (where ","type":"text"},{"text":"","type":"text","marks":[{"type":"code"}]},{"text":" has no periods/dots in it).","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Delete core or error files if there are any. MPAS components will sometimes produce a large number of them. The following commands are useful for checking for these files:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"ls | grep -in core","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"ls | grep -in err","type":"text","marks":[{"type":"code"}]}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"If you are re-submitting the ","type":"text"},{"text":"initial","type":"text","marks":[{"type":"em"}]},{"text":" job, you will need to run ","type":"text"},{"text":"./xmlchange -id CONTINUE_RUN -val TRUE","type":"text","marks":[{"type":"code"}]}]}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Post-Processing with zppy","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"c46b0afb-c9ab-408c-ba91-f82d88302f5b"}}]}]},{"type":"paragraph","content":[{"text":"To post-process a model run, do the following steps. Note that to post-process up to year ","type":"text"},{"text":"n","type":"text","marks":[{"type":"em"}]},{"text":", then you must have short-term archived up to year ","type":"text"},{"text":"n","type":"text","marks":[{"type":"em"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"You can ask questions about ","type":"text"},{"text":"zppy","type":"text","marks":[{"type":"code"}]},{"text":" on ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://github.com/E3SM-Project/zppy/discussions/categories/general"}},{"text":" ","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Install zppy","type":"text"}]},{"type":"paragraph","content":[{"text":"Load the E3SM unified environment. For Chrysalis, this is: ","type":"text"},{"text":"source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh","type":"text","marks":[{"type":"code"}]},{"text":". Commands for other machines, and installation guide for development version of zppy can be found at ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://e3sm-project.github.io/zppy/_build/html/main/getting_started.html"}},{"text":" ","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Configuration File","type":"text"}]},{"type":"paragraph","content":[{"text":"Create a new post-processing configuration file or copy an existing one. A good starting point is the configuration file corresponding to the simulation above: ","type":"text"},{"text":"https://github.com/E3SM-Project/SimulationScripts/blob/master/archive/v2/post.v2.LR.piControl.cfg","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/E3SM-Project/SimulationScripts/blob/master/archive/v2/post.v2.LR.piControl.cfg"}}]},{"text":" (Note: this is a private repo)","type":"text"}]},{"type":"paragraph","content":[{"text":"Call it ","type":"text"},{"text":"post..cfg","type":"text","marks":[{"type":"code"}]},{"text":" and place it in your ","type":"text"},{"text":"","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"Edit the file and customize as needed. The file is structured with ","type":"text"},{"text":"[section]","type":"text","marks":[{"type":"code"}]},{"text":" and ","type":"text"},{"text":"[[sub-sections]]","type":"text","marks":[{"type":"code"}]},{"text":". There is a ","type":"text"},{"text":"[default]","type":"text","marks":[{"type":"code"}]},{"text":" section, followed by additional sections for each available ","type":"text"},{"text":"zppy","type":"text","marks":[{"type":"code"}]},{"text":" task (climo, ts, e3sm_diags, mpas_analysis, …). Sub-sections can be used to have multiple instances of a particular task, for example regridded monthly or globally averaged time series files. Refer to the ","type":"text"},{"text":"zppy documentation","type":"text","marks":[{"type":"link","attrs":{"href":"https://e3sm-project.github.io/zppy/_build/html/main/getting_started.html#configuration-file"}}]},{"text":" for more details.","type":"text"}]},{"type":"paragraph","content":[{"text":"The key sections of the configuration file are:","type":"text"}]},{"type":"paragraph","content":[{"text":"[default]","type":"text","marks":[{"type":"code"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"input","type":"text","marks":[{"type":"code"}]},{"text":", ","type":"text"},{"text":"output","type":"text","marks":[{"type":"code"}]},{"text":", ","type":"text"},{"text":"www","type":"text","marks":[{"type":"code"}]},{"text":" paths may need to be edited.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"[climo]","type":"text","marks":[{"type":"code"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"mapping_file","type":"text","marks":[{"type":"code"}]},{"text":" path may need to be edited.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Typically generate climatology files every 20,50 years: ","type":"text"},{"text":"years = begin_year:end_yr:averaging_period","type":"text","marks":[{"type":"code"}]},{"text":" – e.g., ","type":"text"},{"text":"years = \"1:80:20\", \"1:50:50\",","type":"text","marks":[{"type":"code"}]}]}]}]},{"type":"paragraph","content":[{"text":"[ts]","type":"text","marks":[{"type":"code"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"mapping_file","type":"text","marks":[{"type":"code"}]},{"text":" path may need to be edited.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Time series, typically done in chunks of 10 years – e.g., ","type":"text"},{"text":"years = \"1:80:10\"","type":"text","marks":[{"type":"code"}]}]}]}]},{"type":"paragraph","content":[{"text":"[e3sm_diags]","type":"text","marks":[{"type":"code"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"reference_data_path","type":"text","marks":[{"type":"code"}]},{"text":" may need to be edited.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"short_name","type":"text","marks":[{"type":"code"}]},{"text":" is a shortened version of the ","type":"text"},{"text":"case_name","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"years","type":"text","marks":[{"type":"code"}]},{"text":" should match the ","type":"text"},{"text":"[climo]","type":"text","marks":[{"type":"code"}]},{"text":" section ","type":"text"},{"text":"years","type":"text","marks":[{"type":"code"}]}]}]}]},{"type":"paragraph","content":[{"text":"[mpas_analysis]","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"Years can be specified separately for time series, climatology and ENSO plots. The lists must have the same lengths and each entry will be mapped to a realization of mpas_analysis:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"ts_years = \"1-50\", \"1-100\",","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"enso_years = \"11-50\", \"11-100\",","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"climo_years =\"21-50\", \"51-100\",","type":"text","marks":[{"type":"code"}]}]}]}]},{"type":"paragraph","content":[{"text":"In this particular example, MPAS Analysis will be run twice. The first realization will produce time series plots covering years 1 to 50, ENSO plots for years 11 to 50 and climatology plots averaged over years 21-50. The second realization will cover years 1-100 for time series, 11-100 for ENSO, 51-100 for climatologies.","type":"text"}]},{"type":"paragraph","content":[{"text":"[global_time_series]","type":"text","marks":[{"type":"code"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"ts_years","type":"text","marks":[{"type":"code"}]},{"text":" should match the ","type":"text"},{"text":"[mpas_analysis]","type":"text","marks":[{"type":"code"}]},{"text":" section ","type":"text"},{"text":"ts_years","type":"text","marks":[{"type":"code"}]},{"text":" ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"climo_years","type":"text","marks":[{"type":"code"}]},{"text":" should match the ","type":"text"},{"text":"[mpas_analysis]","type":"text","marks":[{"type":"code"}]},{"text":" section ","type":"text"},{"text":"climo_years","type":"text","marks":[{"type":"code"}]},{"text":" ","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"See the ","type":"text"},{"text":"zppy tutorial","type":"text","marks":[{"type":"link","attrs":{"href":"https://e3sm-project.github.io/zppy/_build/html/main/tutorial.html"}}]},{"text":" for complete configuration file examples.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Launch zppy","type":"text"}]},{"type":"paragraph","content":[{"text":"Make sure you load the E3SM unified environment.","type":"text"}]},{"type":"paragraph","content":[{"text":"Run ","type":"text"},{"text":"zppy -c post..cfg","type":"text","marks":[{"type":"code"}]},{"text":". This will submit a number of jobs. Run ","type":"text"},{"text":"sq","type":"text","marks":[{"type":"code"}]},{"text":" to see what jobs are running.","type":"text"}]},{"type":"paragraph","content":[{"text":"e3sm_diags","type":"text","marks":[{"type":"code"}]},{"text":" jobs are dependent on ","type":"text"},{"text":"climo","type":"text","marks":[{"type":"code"}]},{"text":" and ","type":"text"},{"text":"ts","type":"text","marks":[{"type":"code"}]},{"text":" jobs, so they wait for those to finish. MPAS Analysis jobs re-use computations, so they are chained. Most jobs run quickly, though MPAS Analysis may take several hours.","type":"text"}]},{"type":"paragraph","content":[{"text":"These jobs create a new directory ","type":"text"},{"text":"//post","type":"text","marks":[{"type":"code"}]},{"text":". Each realization will have a shell script (typically bash). This is the actual file that has been submitted to the batch system. There will also be a log file ","type":"text"},{"text":"*.o","type":"text","marks":[{"type":"code"}]},{"text":" as well as a ","type":"text"},{"text":"*.status","type":"text","marks":[{"type":"code"}]},{"text":" file. The status file indicates the state (WAITING, RUNNING, OK, ERROR). Once all the jobs are complete, you can check their status","type":"text"}]},{"type":"codeBlock","content":[{"text":"cd //post/scripts\ncat *.status # should be a list of \"OK\"\ngrep -v \"OK\" *.status # lists files without \"OK\"","type":"text"}]},{"type":"paragraph","content":[{"text":"If you re-run ","type":"text"},{"text":"zppy","type":"text","marks":[{"type":"code"}]},{"text":", it will check status of tasks and will skip any task if its status is “OK”. ","type":"text"}]},{"type":"paragraph","content":[{"text":"As your simulation progresses, you can update the post-processing years in the configuration file and re-run ","type":"text"},{"text":"zppy","type":"text","marks":[{"type":"code"}]},{"text":". Newly added task will be submitted, while previously completed ones will be skipped.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Tasks","type":"text"}]},{"type":"paragraph","content":[{"text":"If you run ","type":"text"},{"text":"ls","type":"text","marks":[{"type":"code"}]},{"text":" you’ll probably see files like ","type":"text"},{"text":"e3sm_diags_180x360_aave_model_vs_obs_0001-0020.status","type":"text","marks":[{"type":"code"}]},{"text":". This is one ","type":"text"},{"text":"e3sm_diags","type":"text","marks":[{"type":"code"}]},{"text":" job. Parts of the file name are explained below:","type":"text"}]},{"type":"table","attrs":{"layout":"default","localId":"9bc98f85-4d03-4833-91ae-eb4bcab1b5d5"},"content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":[340.0]},"content":[{"type":"paragraph","content":[{"text":"Meaning","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":[340.0]},"content":[{"type":"paragraph","content":[{"text":"Part of File Name","type":"text","marks":[{"type":"strong"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[340.0]},"content":[{"type":"paragraph","content":[{"text":"Task","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[340.0]},"content":[{"type":"paragraph","content":[{"text":"e3sm_diags","type":"text","marks":[{"type":"code"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[340.0]},"content":[{"type":"paragraph","content":[{"text":"Grid","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[340.0]},"content":[{"type":"paragraph","content":[{"text":"180x360_aave","type":"text","marks":[{"type":"code"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[340.0]},"content":[{"type":"paragraph","content":[{"text":"Model/obs v. model/obs","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[340.0]},"content":[{"type":"paragraph","content":[{"text":"model_vs_obs","type":"text","marks":[{"type":"code"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[340.0]},"content":[{"type":"paragraph","content":[{"text":"First and last years","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[340.0]},"content":[{"type":"paragraph","content":[{"text":"0001-0020","type":"text","marks":[{"type":"code"}]}]}]}]}]},{"type":"paragraph","content":[{"text":"There is also a corresponding output file. It will have the same name but end with ","type":"text"},{"text":".o","type":"text","marks":[{"type":"code"}]},{"text":" instead of ","type":"text"},{"text":".status","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Output","type":"text"}]},{"type":"paragraph","content":[{"text":"The post-processing output is organized hierarchically as follows (where the exact year-ranges are what are defined in the configuration file):","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"//post/atm/180x360_aave/ts/monthly/10yr","type":"text","marks":[{"type":"code"}]},{"text":" has the time series files – one variable per file, in 10 year chunks as defined in ","type":"text"},{"text":"/post..cfg","type":"text","marks":[{"type":"code"}]},{"text":". ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"//post/atm/180x360_aave/clim/20yr","type":"text","marks":[{"type":"code"}]},{"text":" similarly has climatology files for 20 year periods, as defined in ","type":"text"},{"text":"/post..cfg","type":"text","marks":[{"type":"code"}]},{"text":"`.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"//post/atm/glb/ts/monthly/10yr","type":"text","marks":[{"type":"code"}]},{"text":" has globally averaged files for 10 years periods as defined in ","type":"text"},{"text":"/post..cfg","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Documenting the Model Run","type":"text"}]},{"type":"paragraph","content":[{"text":"You should create a Confluence page for your model run in ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://acme-climate.atlassian.net/wiki/spaces/EWCG/pages/2126938167"}},{"text":". Use ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://acme-climate.atlassian.net/wiki/spaces/EWCG/pages/2297299190"}},{"text":" as a template. See below for how to fill out this template.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Code","type":"text"}]},{"type":"paragraph","content":[{"text":"code_root_dir","type":"text","marks":[{"type":"code"}]},{"text":" and ","type":"text"},{"text":"tag_name","type":"text","marks":[{"type":"code"}]},{"text":" are defined in ","type":"text"},{"text":"/.csh","type":"text","marks":[{"type":"code"}]},{"text":". ","type":"text"}]},{"type":"codeBlock","content":[{"text":"cd /\ngit log","type":"text"}]},{"type":"paragraph","content":[{"text":"The commit hash at the top is the most recent commit.","type":"text"}]},{"type":"paragraph","content":[{"text":"Add “, ” to this section of your page.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Configuration","type":"text"}]},{"type":"paragraph","content":[{"text":"Compset","type":"text","marks":[{"type":"code"}]},{"text":" and ","type":"text"},{"text":"Res","type":"text","marks":[{"type":"code"}]},{"text":" are specified on in the PACE “Experiment Details” section. See “Performance Information” for how to access PACE. Choose the latest job and list these settings on your page.","type":"text"}]},{"type":"paragraph","content":[{"text":"Custom parameters should also be listed. Find these by running:","type":"text"}]},{"type":"codeBlock","content":[{"text":"cd \ngrep -n \"EOF >> user_nl\" run..csh # Find the line numbers to look at","type":"text"}]},{"type":"paragraph","content":[{"text":"Copy the code blocks after ","type":"text"},{"text":"cat <> user_nl_eam","type":"text","marks":[{"type":"code"}]},{"text":", ","type":"text"},{"text":"cat << EOF >> user_nl_elm","type":"text","marks":[{"type":"code"}]},{"text":", and ","type":"text"},{"text":"cat << EOF >> user_nl_mosart","type":"text","marks":[{"type":"code"}]},{"text":" to your page.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Scripts","type":"text"}]},{"type":"paragraph","content":[{"text":"Push your ","type":"text"},{"text":"/run..csh","type":"text","marks":[{"type":"code"}]},{"text":" to ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://github.com/E3SM-Project/SimulationScripts"}},{"text":" , in the ","type":"text"},{"text":"archive/v2/beta/coupled","type":"text","marks":[{"type":"code"}]},{"text":" directory. Then link it to this section of your page.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Output files","type":"text"}]},{"type":"paragraph","content":[{"text":"Specify the path to your output files: ","type":"text"},{"text":"/","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Jobs","type":"text"}]},{"type":"paragraph","content":[{"text":"Fill out a table with columns for “Job”, “Years”, “Nodes”, “SYPD”, and “Notes”.","type":"text"}]},{"type":"paragraph","content":[{"text":"Log file names will give you the job IDs. Logs are found in ","type":"text"},{"text":"//run","type":"text","marks":[{"type":"code"}]},{"text":". If you have done short term archiving, then they will instead be in ","type":"text"},{"text":"//archive/logs","type":"text","marks":[{"type":"code"}]},{"text":". Use ","type":"text"},{"text":"ls","type":"text","marks":[{"type":"code"}]},{"text":" to see what logs are in the directory. The job ID will be the two-part (period-separated) number after ","type":"text"},{"text":".log.","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"PACE’s “Experiment Details” section shows ","type":"text"},{"text":"JobID","type":"text","marks":[{"type":"code"}]},{"text":" as well. In the table, link each job ID to its corresponding PACE web page. Note that failed jobs will not have a web page on PACE, but you should still list them in the table.","type":"text"}]},{"type":"paragraph","content":[{"text":"Use ","type":"text"},{"text":"zgrep \"DATE=\" | head -n 1","type":"text","marks":[{"type":"code"}]},{"text":" to find the start date. Use ","type":"text"},{"text":"zgrep \"DATE=\" | tail -n 1","type":"text","marks":[{"type":"code"}]},{"text":" to find the end date. If you would like you can write a ","type":"text"},{"text":"bash","type":"text","marks":[{"type":"code"}]},{"text":" function to make this easier:","type":"text"}]},{"type":"codeBlock","content":[{"text":"get_dates()\n{\n for f in atm.log.*.gz; do\n echo $f\n zgrep \"DATE=\" $f | head -n 1\n zgrep \"DATE=\" $f | tail -n 1\n\techo \"\"\n done\n}","type":"text"}]},{"type":"paragraph","content":[{"text":"(If ","type":"text"},{"text":"zgrep","type":"text","marks":[{"type":"code"}]},{"text":" is unavailable, use ","type":"text"},{"text":"less ","type":"text","marks":[{"type":"code"}]},{"text":" to look at a gzipped log file. Scroll down a decent amount to ","type":"text"},{"text":"DATE=","type":"text","marks":[{"type":"code"}]},{"text":" to find the start date. Use ","type":"text"},{"text":"SHIFT+g","type":"text","marks":[{"type":"code"}]},{"text":" to go to the end of the file. Scroll up to ","type":"text"},{"text":"DATE=","type":"text","marks":[{"type":"code"}]},{"text":" to find the end date.)","type":"text"}]},{"type":"paragraph","content":[{"text":"In the “Years” column specify ","type":"text"},{"text":" - ","type":"text","marks":[{"type":"code"}]},{"text":", with each in ","type":"text"},{"text":"year-month-day","type":"text","marks":[{"type":"code"}]},{"text":" format.","type":"text"}]},{"type":"paragraph","content":[{"text":"To find the number of nodes, first look at the Processor # / Simulation Time chart on PACE.","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"6b634799-a8f0-4cad-8097-86c201021485"}}]},{"text":" The x-axis lists the highest MPI rank used, with base-0 numbering of ranks. (PE layouts often don’t fit exactly N nodes but instead fill N-1 nodes and have some number of ranks left over on the final node, leaving some cores on that node unused.) Then find ","type":"text"},{"text":"MPI tasks/node","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"6b634799-a8f0-4cad-8097-86c201021485"}},{"type":"code"}]},{"text":" in the “Experiment Details” section","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"6b634799-a8f0-4cad-8097-86c201021485"}}]},{"text":". The number of nodes can then be calculated as ","type":"text"},{"text":"ceil((highest MPI rank + 1)/ (MPI tasks/node))","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"The SYPD (simulated years per day) is listed in PACE’s “Experiment Details” section as ","type":"text"},{"text":"Model Throughput","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"In the “Notes” section, mention if a job failed or if you changed anything before re-running a job.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Global time series","type":"text"}]},{"type":"paragraph","content":[{"text":"Be sure to have set the ","type":"text"},{"text":"[global_time_series]","type":"text","marks":[{"type":"code"}]},{"text":" task in ","type":"text"},{"text":"zppy","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"Example configuration:","type":"text"}]},{"type":"codeBlock","content":[{"text":"# Global time series plots\n[global_time_series]\nactive = False\nyears = \"1-100\",\nts_num_years = 10\nfigstr=coupled_v2_test01\nmoc_file=mocTimeSeries_0001-0100.nc\nexperiment_name=20210409.v2beta4.piControl.ne30pg2_EC30to60E2r2.chrysalis\nts_years = \"1-50\", \"1-100\",","type":"text"}]},{"type":"paragraph","content":[{"text":"That will produce ","type":"text"},{"text":".pdf","type":"text","marks":[{"type":"code"}]},{"text":" and ","type":"text"},{"text":".png","type":"text","marks":[{"type":"code"}]},{"text":". They will be available automatically at ","type":"text"},{"text":"//.png","type":"text","marks":[{"type":"code"}]},{"text":". You can download the image from the website and then upload it to your Confluence page.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"E3SM Diags","type":"text"}]},{"type":"paragraph","content":[{"text":"The template page already includes baseline diagnostics. Add your own diagnostics links labeled as ","type":"text"},{"text":"-","type":"text","marks":[{"type":"code"}]},{"text":". ","type":"text"}]},{"type":"paragraph","content":[{"text":"Your diagnostics are located at the web address corresponding to the ","type":"text"},{"text":"www","type":"text","marks":[{"type":"code"}]},{"text":" path in ","type":"text"},{"text":"/.cfg","type":"text","marks":[{"type":"code"}]},{"text":". ","type":"text"}]},{"type":"paragraph","content":[{"text":"See ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://e3sm-project.github.io/e3sm_diags/_build/html/master/quickguides/quick-guide-general.html"}},{"text":" for finding the URLs for the web portals on each E3SM machine (listed as ","type":"text"},{"text":"","type":"text","marks":[{"type":"code"}]},{"text":"). ","type":"text"}]},{"type":"paragraph","content":[{"text":"Fill the table with the specific web links: e.g., ","type":"text"},{"text":"https://web.lcrc.anl.gov/public/e3sm/diagnostic_output//E3SM/v2/beta//e3sm_diags/180x360_aave/model_vs_obs_0001-0020/viewer/","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"MPAS Analysis","type":"text"}]},{"type":"paragraph","content":[{"text":"See ","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"79611877-fbb0-472a-97aa-7959191531d3"}}]},{"type":"inlineCard","attrs":{"url":"https://e3sm-project.github.io/e3sm_diags/_build/html/master/quickguides/quick-guide-general.html"}},{"text":" for finding the URLs for the web portals on each E3SM machine (listed as ","type":"text"},{"text":"","type":"text","marks":[{"type":"code"}]},{"text":")","type":"text"}]},{"type":"paragraph","content":[{"text":"Make a bulleted list of links, e.g., for ","type":"text"},{"text":"https://web.lcrc.anl.gov/public/e3sm/diagnostic_output//E3SM/v2/beta//mpas_analysis/ts_0001-0050_climo_0021-0050/","type":"text","marks":[{"type":"code"}]},{"text":", create a bullet “1-50 (time series), 21-50 (climatology)”.","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Long Term Archiving with zstash","type":"text"}]},{"type":"paragraph","content":[{"text":"Simulations that are deemed sufficiently valuable should be archived using ","type":"text"},{"text":"zstash","type":"text","marks":[{"type":"code"}]},{"text":" for long-term preservation.","type":"text"}]},{"type":"paragraph","content":[{"text":"You can ask questions about ","type":"text"},{"text":"zstash","type":"text","marks":[{"type":"code"}]},{"text":" on ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://github.com/E3SM-Project/zstash/discussions/categories/general"}},{"text":" ","type":"text"}]},{"type":"paragraph","content":[{"text":"Compy, Anvil and Chrysalis do not have local HPSS. We rely on NERSC HPSS for long-term archiving. ","type":"text"}]},{"type":"paragraph","content":[{"text":"If you are archiving a simulation run on Compy or LCRC (Chrysalis/Anvil), do all of the following steps. If you are archiving a simulation run on NERSC (Cori), skip to step 4.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"1. Clean up directory","type":"text"}]},{"type":"paragraph","content":[{"text":"Log into the machine that you ran the simulation on.","type":"text"}]},{"type":"paragraph","content":[{"text":"Remove all ","type":"text"},{"text":"eam.i","type":"text","marks":[{"type":"code"}]},{"text":" files except the latest one. Dates are of the form ","type":"text"},{"text":"","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ cd //run\n$ ls | wc -l # See how many items are in this directory\n$ mv .eam.i.-00000.nc tmp.nc\n$ rm .eam.i.*.nc\n$ mv tmp.nc .eam.i.2015-01-01-00000.nc\n$ ls | wc -l # See how many items are in this directory","type":"text"}]},{"type":"paragraph","content":[{"text":"There may still be more files than is necessary to archive. You can probably remove ","type":"text"},{"text":"*.err","type":"text","marks":[{"type":"code"}]},{"text":", ","type":"text"},{"text":"*.lock","type":"text","marks":[{"type":"code"}]},{"text":", ","type":"text"},{"text":"*debug_block*","type":"text","marks":[{"type":"code"}]},{"text":", ","type":"text"},{"text":"*ocean_block_stats*","type":"text","marks":[{"type":"code"}]},{"text":" files.","type":"text"}]},{"type":"paragraph"},{"type":"heading","attrs":{"level":2},"content":[{"text":"2. ","type":"text"},{"text":"zstash create","type":"text","marks":[{"type":"code"}]},{"text":" & Transfer to NERSC HPSS","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"2.a. E3SM Unified v1.6.0 / zstash v1.2.0 or greater","type":"text"}]},{"type":"paragraph","content":[{"text":"If you are using ","type":"text"},{"text":"E3SM Unified v1.6.0","type":"text","marks":[{"type":"code"}]},{"text":" or greater, ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://github.com/E3SM-Project/zstash/pull/154"}},{"text":" has enabled Globus (","type":"text"},{"type":"inlineCard","attrs":{"url":"https://www.globus.org/"}},{"text":" ) transfer with ","type":"text"},{"text":"zstash create","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"On the machine that you ran the simulation on:","type":"text"}]},{"type":"paragraph","content":[{"text":"If you don’t have one already, create a directory for utilities, e.g., ","type":"text"},{"text":"utils","type":"text","marks":[{"type":"code"}]},{"text":". Then open a file in that directory called ","type":"text"},{"text":"batch_zstash_create.bash","type":"text","marks":[{"type":"code"}]},{"text":" and paste the following in it, making relevant edits:","type":"text"}]},{"type":"codeBlock","content":[{"text":"#!/bin/bash\n\n# Run on \n\n# Load E3SM Unified\n\n\n# List of experiments to archive with zstash\nEXPS=(\\\n \\\n)\n\n# Loop over simulations\nfor EXP in \"${EXPS[@]}\"\ndo\n echo === Archiving ${EXP} ===\n cd /${EXP}\n mkdir -p zstash\n stamp=`date +%Y%m%d`\n time zstash create -v --hpss=globus://nersc/home///E3SMv2/${EXP} --maxsize 128 . 2>&1 | tee zstash/zstash_create_${stamp}.log\ndone","type":"text"}]},{"type":"paragraph","content":[{"text":"Commands to load the E3SM Unified environment for each machine can be found at ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://e3sm-project.github.io/zppy/_build/html/main/getting_started.html"}},{"text":" .","type":"text"}]},{"type":"paragraph","content":[{"text":"Then, do the following:","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ screen # Enter screen\n$ screen -ls # Output should say \"Attached\"\n$ ./batch_zstash_create.bash 2>&1 | tee batch_zstash_create.log\n# Control A D to exit screen\n# DO NOT CONTROL X / CONTROL C (as for emacs). This will terminate the task running in screen!!!\n\n$ screen -ls # Output should say \"Detached\"\n$ hostname\n# If you log in on another login node, \n# then you will need to ssh to this one to get back to the screen session.\n$ tail -f batch_zstash_create.log # Check log without going into screen\n# Wait for this to finish\n\n$ screen -r # Return to screen\n# Check that output ends with `real`, `user`, `sys` time information\n$ exit # Terminate screen\n\n$ screen -ls # The screen should no longer be listed\n$ ls //zstash\n# `index.db`, and a `zstash_create` log should be present\n# No tar files should be listed\n\n# If you'd like to know how much space the archive or entire simulation use, run:\n$ du -sh //zstash\n$ du -sh /","type":"text"}]},{"type":"paragraph","content":[{"text":"Then, on NERSC/Cori:","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ hsi\n$ ls /home///E3SMv2/\n# Tar files and `index.db` should be listed.\n# Note `| wc -l` doesn't work on hsi\n$ exit","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"2.b. Earlier releases","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"2.b.i. ","type":"text"},{"text":"zstash create","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"On the machine that you ran the simulation on:","type":"text"}]},{"type":"paragraph","content":[{"text":"If you don’t have one already, create a directory for utilities, e.g., ","type":"text"},{"text":"utils","type":"text","marks":[{"type":"code"}]},{"text":". Then open a file in that directory called ","type":"text"},{"text":"batch_zstash_create.bash","type":"text","marks":[{"type":"code"}]},{"text":" and paste the following in it, making relevant edits:","type":"text"}]},{"type":"codeBlock","content":[{"text":"#!/bin/bash\n\n# Run on \n\n# Load E3SM Unified\n\n\n# List of experiments to archive with zstash\nEXPS=(\\\n \\\n)\n\n# Loop over simulations\nfor EXP in \"${EXPS[@]}\"\ndo\n echo === Archiving ${EXP} ===\n cd /${EXP}\n mkdir -p zstash\n stamp=`date +%Y%m%d`\n time zstash create -v --hpss=none --maxsize 128 . 2>&1 | tee zstash/zstash_create_${stamp}.log\ndone","type":"text"}]},{"type":"paragraph","content":[{"text":"Commands to load the E3SM Unified environment for each machine can be found at ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://e3sm-project.github.io/zppy/_build/html/main/getting_started.html"}},{"text":" .","type":"text"}]},{"type":"paragraph","content":[{"text":"Then, do the following:","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ screen # Enter screen\n$ screen -ls # Output should say \"Attached\"\n$ ./batch_zstash_create.bash 2>&1 | tee batch_zstash_create.log\n# Control A D to exit screen\n# DO NOT CONTROL X / CONTROL C (as for emacs). This will terminate the task running in screen!!!\n\n$ screen -ls # Output should say \"Detached\"\n$ hostname\n# If you log in on another login node, \n# then you will need to ssh to this one to get back to the screen session.\n$ tail -f batch_zstash_create.log # Check log without going into screen\n# Wait for this to finish\n# (On Chrysalis, for 165 years of data, this takes ~14 hours)\n\n$ screen -r # Return to screen\n# Check that output ends with `real`, `user`, `sys` time information\n$ exit # Terminate screen\n\n$ screen -ls # The screen should no longer be listed\n$ ls //zstash\n# tar files, `index.db`, and a `zstash_create` log should be present\n\n# If you'd like to know how much space the archive or entire simulation use, run:\n$ du -sh //zstash\n$ du -sh /","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"2.b.ii. Transfer to NERSC","type":"text"}]},{"type":"paragraph","content":[{"text":"On a NERSC machine (Cori):","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ mkdir -p /global/cfs/cdirs/e3sm//E3SMv2/\n$ ls /global/cfs/cdirs/e3sm//E3SMv2/\n# Should be empty","type":"text"}]},{"type":"paragraph","content":[{"text":"Log into Globus, using your NERSC credentials: ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://www.globus.org/"}},{"text":" ","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"(Left hand side): Transfer from ","type":"text"},{"text":" //zstash","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Click enter on path and \"select all\" on left-hand side","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Transfer to ","type":"text"},{"text":"NERSC DTN /global/cfs/cdirs/e3sm//E3SMv2/","type":"text","marks":[{"type":"code"}]}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Notice we're using ","type":"text"},{"text":"cfs","type":"text","marks":[{"type":"code"}]},{"text":" rather than ","type":"text"},{"text":"scratch","type":"text","marks":[{"type":"code"}]},{"text":" on Cori","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Click enter on the path","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Click “Transfer & Sync Options” in the center.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Choose:","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"“sync - only transfer new or changed files” (choose “modification time is newer” in the dropdown box)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"“preserve source file modification times”","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"“verify file integrity after transfer”","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"For “Label This Transfer”: “zstash to NERSC”","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Click \"Start\" on the left hand side.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"You should get an email from Globus when the transfer is completed. (On Chrysalis, for 165 years of data, this transfer takes ~13 hours).","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"2.b.iii. Transfer to HPSS","type":"text"}]},{"type":"paragraph","content":[{"text":"On a NERSC machine (Cori):","type":"text"}]},{"type":"codeBlock","content":[{"text":"Log in to Cori\n$ cd /global/cfs/cdirs/e3sm/forsyth//\n$ ls | wc -l\n# Should match the number of files in the other machine's `//zstash`\n$ ls *.tar | wc -l\n# Should be two less than the previous result, \n# since `index.db` and the `zstash_create` log are also present.\n\n$ hsi\n$ pwd\n# Should be /home//\n$ ls E3SMv2\n# Check what you already have in the directory\n# You don't want to accidentally overwrite a directory already in HPSS.\n$ exit\n\n$ cd /global/cfs/cdirs/e3sm//E3SMv2\n\n$ screen\n$ screen -ls # Output should say \"Attached\"\n# https://www2.cisl.ucar.edu/resources/storage-and-file-systems/hpss/managing-files-hsi\n# cput will not transfer file if it exists.\n$ hsi \"cd /home///E3SMv2/; cput -R \"\n# Control A D to exit screen\n# DO NOT CONTROL X / CONTROL C (as for emacs). This will terminate the task running in screen!!!\n\n$ screen -ls # Output should say \"Detached\"\n$ hostname\n# If you log in on another login node, \n# then you will need to ssh to this one to get back to the screen session.\n# Wait for the `hsi` command to finish\n# (On Chrysalis, for 165 years of data, this takes ~2 hours)\n\n$ screen -r # Return to screen\n# Check output for any errors\n$ exit # Terminate screen\n\n$ screen -ls # The screen should no longer be listed\n$ hsi\n$ ls /home///E3SMv2/\n# Should match the number of files in the other machine's `//zstash`\n$ exit","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"3. ","type":"text"},{"text":"zstash check","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"655f27d5-6b06-4fc7-af94-a5135ce838c5"}},{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"On a NERSC machine (Cori):","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ cd /global/homes//\n$ emacs batch_zstash_check.bash","type":"text"}]},{"type":"paragraph","content":[{"text":"Paste the following in that file, making relevant edits:","type":"text"}]},{"type":"codeBlock","content":[{"text":"#!/bin/bash\n\n# Run on NERSC dtn\n\n# Load environment that includes zstash\n\n\n# List of experiments to archive with zstash\nEXPS=(\\\n \\\n)\n\n# Loop over simulations\nfor EXP in \"${EXPS[@]}\"\ndo\n echo === Checking ${EXP} ===\n cd /global/cfs/cdirs/e3sm//E3SMv2\n mkdir -p ${EXP}/zstash\n cd ${EXP}\n stamp=`date +%Y%m%d`\n time zstash check -v --hpss=/home///E3SMv2/${EXP} --workers 2 2>&1 | tee zstash/zstash_check_${stamp}.log\ndone","type":"text"}]},{"type":"paragraph","content":[{"text":"Commands to load the E3SM Unified environment for each machine can be found at ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://e3sm-project.github.io/zppy/_build/html/main/getting_started.html"}},{"text":" .","type":"text"}]},{"type":"panel","attrs":{"panelType":"note"},"content":[{"type":"paragraph","content":[{"text":"If you’re using E3SM Unified v1.6.0 / zstash v1.2.0 (or greater) and you want to check a long simulation, you can use the ","type":"text"},{"text":"--tars","type":"text","marks":[{"type":"code"}]},{"text":" option introduced in ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://github.com/E3SM-Project/zstash/pull/170"}},{"text":" to split the checking into more manageable pieces:","type":"text"}]},{"type":"paragraph","content":[{"text":"# Starting at 00005a until the end\nzstash check --tars=00005a-\n# Starting from the beginning to 00005a (included)\nzstash check --tars=-00005a\n# Specific range\nzstash check --tars=00005a-00005c\n# Selected tar files\nzstash check --tars=00003e,00004e,000059\n# Mix and match\nzstash check --tars=000030-00003e,00004e,00005a-","type":"text"}]}]},{"type":"paragraph","content":[{"text":"Then, do the following:","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ ssh dtn01.nersc.gov\n\n$ screen\n$ screen -ls # Output should say \"Attached\"\n$ cd /global/homes//\n$ ./batch_zstash_check.bash\n# Control A D to exit screen\n# DO NOT CONTROL X / CONTROL C (as for emacs). This will terminate the task running in screen!!!\n\n$ screen -ls # Output should say \"Detached\"\n$ hostname\n# If you log in on another login node, \n# then you will need to ssh to this one to get back to the screen session.\n# Wait for the script to finish\n# (On Chrysalis, for 165 years of data, this takes ~5 hours)\n\n$ screen -r # Return to screen\n# Check that output ends with `INFO: No failures detected when checking the files.`\n# as well as listing real`, `user`, `sys` time information\n$ exit # Terminate screen\n\n$ screen -ls # The screen should no longer be listed\n$ exit # exit data transfer node\n$ cd /global/cfs/cdirs/e3sm//E3SMv2//zstash\n$ tail zstash_check_.log\n# Output should match the output from the screen (without the time information)","type":"text"}]},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"Because of ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://github.com/E3SM-Project/zstash/issues/167"}},{"text":" , for now it is a good idea to run ","type":"text"},{"text":"grep -i Exception zstash_check_.log","type":"text","marks":[{"type":"code"}]},{"text":" to confirm success.","type":"text"}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"4. Document","type":"text"}]},{"type":"paragraph","content":[{"text":"On a NERSC machine (Cori):","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ hsi\n$ ls /home///E3SMv2\n# Check that the simulation case is now listed in this directory\n$ ls /home///E3SMv2/\n# Should match the number of files in the other machine's `//zstash`\n$ exit\n\n$ cd /global/cfs/cdirs/e3sm//E3SMv2//zstash\n$ ls\n# `index.db` and `zstash_check` log should be the only items listed\n# https://www2.cisl.ucar.edu/resources/storage-and-file-systems/hpss/managing-files-hsi\n# cput will not transfer file if it exists.\n$ hsi \"cd /home///E3SMv2/; cput -R \"\n\n$ hsi\n$ ls /home///E3SMv2/\n# tar files, `index.db`, `zstash_create` log, and `zstash_check` log should be present\n$ exit","type":"text"}]},{"type":"paragraph","content":[{"text":"Update simulation Confluence page with information regarding this simulation (For Water Cycle’s v2 work, that page is ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://acme-climate.atlassian.net/wiki/spaces/ED/pages/2766340117"}},{"text":" ) . In the ","type":"text"},{"text":"zstash archive","type":"text","marks":[{"type":"code"}]},{"text":" column, specify: ","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"/home///E3SMv2/","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"zstash_create_.log","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"zstash_check_.log","type":"text","marks":[{"type":"code"}]}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"5. Delete files","type":"text"}]},{"type":"paragraph","content":[{"text":"On a NERSC machine (Cori):","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ hsi\n$ ls /home///E3SMv2/\n# tar files, `index.db`, `zstash_create` log, and `zstash_check` log should be present\n# So, we can safely delete these items on cfs\n$ exit\n\n$ ls /global/cfs/cdirs/e3sm//E3SMv2/\n# Should match output from the `ls` above\n$ cd /global/cfs/cdirs/e3sm/\n$ ls E3SMv2\n# Only the you just transferred to HPSS should be listed\n$ rm -rf E3SMv2","type":"text"}]},{"type":"paragraph","content":[{"text":"On the machine that you ran the simulation on:","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ cd /\n$ ls zstash\n# tar files, index.db, `zstash_create` log should be present\n$ rm -rf zstash # Remove the zstash directory, keeping original files\n$ cd ","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"More info","type":"text"}]},{"type":"paragraph","content":[{"text":"Refer to ","type":"text"},{"text":"zstash's best practices for E3SM","type":"text","marks":[{"type":"link","attrs":{"href":"https://e3sm-project.github.io/zstash/_build/html/master/best_practices.html"}}]},{"text":" for details.","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Publishing the simulation data (Optional)","type":"text"}]},{"type":"paragraph","content":[{"text":"E3SM Project has policy to publish all official simulation campaigns once those simulations are documented in publications. Refer to step 3 in ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/1159594096"}},{"text":" for guidance on requesting data publication.","type":"text"}]}],"version":1}