Day 2- MPAS-Analysis Practicum

• copy the template.mpas_analysis_short.cfg into a new file my_exercise1.cfg. Open it with your editor of choice.

The config file is split into different sections defined by section title denoted by [] (such as [run] or [execute]).

• specify your own output directory:

update [output]/baseDirectory to your own directory /pscratch/sd/<first-letter-of-moniker>/<moniker>/e3sm2024tutorial/mpas-analysis/exercise1

and htmlSubdirectory to /global/cfs/cdirs/ntrain6/www/<moniker>/e3sm2024tutorial/exercise1

• read the [generate] line. This line specifies the analysis tasks that will be performed.

• read the date ranges (defined by startYear and endYear). Note that these appear twice, once in a section under [climatology]and once under [timeSeries].

• make sure you save your changes.

Q1.a. Describe in your own words the tasks and plots you expect this script to produce.

• open job_script_template.slurm

• read and check the #SBATCH options at the top of the file. Key options for today are --account=ntrain6, --reservation=e3sm_day2, --time=00:10:00.

• modify the mpas-analysis command to point to your my_exercise1.cfg (replacing template.mpas_analysis_short.cfg)

• in the directory, run the following command: Note the job ID generated

sbatch job_script_template.slurm

You can check the job status by running:

squeue -u $USER

The analysis script should take approx. 2min to run once it starts. It may spend longer in the queue before starting. (If queue wait times are long, we can switch to the debug queue)

cd $CFS/ntrain6/www/
chmod -R o+rx $USER

Q1.b. How would you know that the job is done? how would you know if it was done successfully?

• open the html directory in your favorite browser: https://portal.nersc.gov/cfs/ntrain6/<moniker>/e3sm2024tutorial/exercise1

Option 1: if the page does not exist, or is empty, double-check that the html path is consistent with your config file, then proceed to the Troubleshooting section.

Option 2: if the page exists and the plots are present, congratulations! Navigate the available plots and answer the following questions.

Q2.a. Navigate the available plots (you can click on plots). Note which plots are available and compare to your answer to Q1.a. Is there anything that you had not anticipated?

Q2.b. Summarize the key features that you notice (either positive or negative) in the ocean and sea ice analysis plots produced. Make sure to take note of the plots axis labels, title and legend.

Q2.c. Pick one concerning feature you noted in Q2.b. How would you go about determining whether this signal is significant? How would you go about determining whether this feature is new?

• copy the template.mpas_analysis_short.cfg into a new config file my_exercise2.cfg

• modify the output directories (both the baseDirectory and htmlDirectory)

• make one or two modifications in the tasks. We suggest:

  • changing the year range in either climatology, timeseries, or both. (note that the available range is 2000-2014, go beyond that at your own peril!)

  • adding tasks, such as indexNino34, climatologyMapSchmidtko, or streamfunctionMOC to the generate line.

You can find the list of tasks by running mpas_analysis --list in the terminal window (provided the e3sm-unified environment has been loaded). Some tasks are more time-consuming (e.g. regional profiles, regional time series) and will not be able to run within this 1hr tutorial. If attempting more time-consuming tasks, make sure to update the wall time and/or nodes in your job script (e.g. --time in job_script_template.cfg).

• modify the mpas-analysis command in job_script_template.slurm to point to your my_exercise2.cfg. Submit by running sbatch job_script_template.slurm.

• Once the job is done, open the html directory in your browser.

Option 1: if the page does not exist, or is empty, double-check that the html path is consistent with your config file, then proceed to the Troubleshooting section.

Option 2: if the page exists and the plots are present, congratulations! Navigate the available plots and answer the following questions.

Q3.a. What additional plots are present? Note any additional features or differences compared to Q2.b.

• Beyond the plots produced, the post-processed output is available for your own analysis. In the terminal window, navigate to the analysis directory (the path is specified in your [output]/baseDirectory) and locate some example output files (.nc files):

Q3.b. Find the post-processed seasonal climatology of the ocean mixed layer depth in January-February-March. What is the full path to the relevant netcdf file?

Q3.c. If you included the MOC task, find the longest post-processed timeseries of the meridional overturning circulation (“moc”). What is the full path to the relevant netcdf file?

• If you have not encountered any technical issues, take time to go through the Troubleshooting section to familiarize yourself with logs.

• Choose between the 2 options below, and open the relevant link.

Option 1: standard MPAS-Analysis output

Option2: enhanced output with polar focus

Q4.a First, identify the simulation and period(s) analyzed. What additional plots are present (compared to Exercises 1-2 above)? Are there any that are missing?

Q4.b Find the .cfg file. (hint: check the html landing page). What are the main differences between this config file and the ones you used in the previous exercises? (you can use vimdiff if you have local copies of both scripts).

Q4.c Which lines would you modify to reproduce this analysis for the run provided in this tutorial? Do you expect to modify other files? (hint: consider why we are not running this analysis now)

• Open this example of MPAS-Analysis output

Q4.a First, identify the simulation(s) and period(s) analyzed. What are the main differences between these plots and the ones shown previously?

Q4.b Find the .cfg file. (hint: check the html landing page). What are the main differences between this config file and the ones you used in the previous exercises? (you can use vimdiff if you have local copies of both scripts).

Q4.c Find the complete .cfg file. (hint: check the html landing page). In the plots provided, the primary simulation is plotted in black, and the control simulation is plotted in red. Which lines would you modify to switch the color of the control to black, and the primary simulation result to red?

• Is the html path correct?
• Was the job submitted?

If an error occurred at submission, double-check the options and commands in the job_script. Note the job ID if it got successfully submitted.

• Did the job finish successfully?

You can check the job history and status by running the command

sacct --starttime 2024-05-08 --format=User,JobID,Jobname,partition,state,time,start,end,elapsed,nnodes

Check the mpas_analysis.o<jobID> file for any runtime information. If it exists, check the mpas_analysis.e<jobID> file for any error information.

• Check the logs in baseDirectory/logs

Each task has produced a log. The taskProgress.log shows the progress through the tasks.

In the log directory, you can search quickly for an error by running the command

 grep -i error *.log

• You can rerun the job with options (e.g. mpas-analysis --purge --verbose my_config.cfg) to provide additional log information.

Alternatively, you can choose to run mpas-analysis on an interactive job. To try this out, use the following commands:

salloc -A ntrain6 --reservation=e3sm_day2 -C cpu -N 1 -c 32 -t 30 -q shared

Once the interactive job is active, run the following commands:

export OMP_NUM_THREADS=1
source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh
export HDF5_USE_FILE_LOCKING=FALSE
vim template.mpas_analysis.cfg
mpas_analysis template.mpas_analysis.cfg

You will see updates on progress in the terminal window, e.g.:

Detected E3SM supported machine: pm-cpu
Using the following config files:
...
Running tasks: 100% |################################################################################################################| Time:  0:01:44
Log files for executed tasks can be found in <path>/logs
Total setup time: 0:00:32.58
Total run time: 0:02:22.48
Generating webpage for viewing results...
Web page: https://portal.nersc.gov/cfs/e3sm/<html-path>/interactive/

You can run MPAS-Analysis within zppy (will be covered on later days).

This is done by adjusting two main section in the zppy configuration file. Default include the input and output paths, mpas_analysis contains details for the analysis.

[default]
input = /pscratch/sd/t/terai/E3SM_simulation/extendedOutput.v3.LR.historical_0101
output = /pscratch/sd/a/abarthel/data/E3SMv3/20240429.tutorial.zppy
case = v3.LR.historical
www = /global/cfs/cdirs/e3sm/www/abarthel/tutorial/20240429.tutorial.zppy 
partition = pm-cpu
environment_commands = "source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh"

[mpas_analysis]
active = True
walltime = "01:00:00"
parallelTaskCount = 6
anomalyRefYear = 2000
ts_years = "2000-2004",
enso_years = "2000-2004",
climo_years = "2000-2004",
mesh = "IcoswISC30E3r5"
shortTermArchive = True
purge = True

To run the analysis, use the following commands in the terminal window:

source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh
zppy -c my_zppy.cfg

Troubleshooting:

• navigate to the <outputdirectory>/post/scripts
• check the status file(s) (you can use grep -i error *.status if you ran many tasks)
• check the .o and .e files for additional information
• Log files for executed tasks can be found in <outputdirectory>/analysis/mpas_analysis/<task>/logs