IN PROGRESS

Contact: Ryan Forsyth

Useful Aliases

Setting some aliases may be useful in running the model. You can edit ~/.bashrc to add aliases. Run source ~/.bashrc to start using them.

...

<code_source_dir>: ${HOME}/E3SM/code<post_processing_script_dir>: ${HOME}/E3SM/utils/post_v2. You might choose instead to use the same directory as <code_source_dir>.

<simulations_dir>: /lcrc/group/e3sm/<username>/E3SM_simulations

...

Performance Information

...

To gzip log files from failed jobs, run gzip *.log.<job ID>.*

Post-Processing with zppy

...

To post-process a model run, do the following steps. Note that to post-process up to year n, then you must have short-term archived up to year n.

cd <post_processing_script_dir>

Configuration File

Create a new configuration file or copy an existing one. Call it <case_name>.cfg.

The sections of the configuration file are described below:

[default]

• input, output, www paths may need to be edited.

[climo]

• mapping_file path may need to be edited.

• Typically generate climatology files every 20,50 years: years = begin_year:end_yr:averaging_period – e.g., years = "1:80:20", "1:50:50",

[ts]

• mapping_file path may need to be edited.

• Time series, typically done in chunks of 10 years – e.g., years = "1:80:10"

[e3sm_diags]

...

Install zppy

If you haven't already, check out the zppy repo in <code_source_dir>. Go to https://github.com/E3SM-Project/zppy . Get path to clone by clicking the green "Code" button. Run git clone <path>.

Load the E3SM unified environment. For LCRC machines, this is: source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified.sh. Commands for other machines can be found at https://e3sm-project.github.io/e3sm_diags/docs/html/quickguides/quick-guide-general.html.

Configuration File

Create a new post-processing configuration file or copy an existing one. A good starting point is the configuration file corresponding to the simulation above: post.20210409.v2beta4.piControl.ne30pg2_EC30to60E2r2.chrysalis.cfg

Call it post.<case_name>.cfg and place it in your <run_scripts_dir>.

Edit the file and customize as needed. The file is structured with [section] and [[sub-sections]]. There is a [default] section, followed by additional sections for each available zppy task (climo, ts, e3sm_diags, mpas_analysis, …). Sub-sections can be used to have multiple instances of a particular task, for example regridded monthly or globally averaged time series files.

The key sections of the configuration file are:

[default]

• input, output, www paths may need to be edited.

[climo]

• mapping_file path may need to be edited.

• Typically generate climatology files every 20,50 years: years = begin_year:end_yr:averaging_period – e.g., years = "1:80:20", "1:50:50",

[ts]

• mapping_file path may need to be edited.

• Time series, typically done in chunks of 10 years – e.g., years = "1:80:10"

[e3sm_diags]

• reference_data_path may need to be edited.

• short_name is a shortened version of the case_name

• years should match the [climo] section years

[mpas_analysis]

• enso_years should start with year 11 -- e.g., enso_years = "11-50",

Launch Post-Processing Jobs

If you haven't already, check out the zppy repo in <post_processing_script_dir>. Go to https://github.com/E3SM-Project/zppy . Get path to clone by clicking the green "Code" button. Run git clone <path>.

Load the E3SM unified environment. For LCRC machines, this is: source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified.sh. Commands for other machines can be found at https://e3sm-project.github.io/e3sm_diags/docs/html/quickguides/quick-guide-general.html.

Run python zppy/post.py -c <case_name>.cfg. This will launch a number of jobs. Run sq to see what jobs are running.

e3sm_diags jobs are dependent on climo and ts jobs, so they wait for those to finish. Most run quickly, though MPAS analysis may take several hours.

...

• _name

• years should match the [climo] section years

[mpas_analysis]

Years can be specified separately for time series, climatology and ENSO plots. The lists must have the same lengths and each entry will be mapped to a realization of mpas_analysis:

• ts_years = "1-50", "1-100",

• enso_years = "11-50", "11-100",

• climo_years ="21-50", "51-100",

In this particular example, MPAS Analysis will be run twice. The first realization will produce time series plots covering years 1 to 50, ENSO plots for years 11 to 50 and climatology plots averaged over years 21-50. The second realization will cover years 1-100 for time series, 11-100 for ENSO, 51-100 for climatologies.

Launch zppy

Make sure you load the E3SM unified environment.

Run python <code_source_dir>/zppy/post.py -c post.<case_name>.cfg. This will submit a number of jobs. Run sq to see what jobs are running.

e3sm_diags jobs are dependent on climo and ts jobs, so they wait for those to finish. MPAS Analysis jobs re-use computations, so they are chained. Most jobs run quickly, though MPAS Analysis may take several hours.

These jobs create a new directory <simulations_dir>/<case_name>/post. Each realization will have a shell script (typically bash). This is the actual file that has been submitted to the batch system. There will also be a log file *.o<job ID> as well as a *.status file. The status file indicates the state (WAITING, RUNNING, OK, ERROR). Once all the jobs are complete, you can check their status

cd <simulations_dir>/<case_name>/post/scripts
cat *.status # should be a list of "OK"
grep -v "OK" *.status # lists files without "OK"

If you re-run post-processing, it will check status of tasks and will skip a task if its status is “OK”zppy, it will check status of tasks and will skip any task if its status is “OK”.

As your simulation progresses, you can update the post-processing years in the configuration file and re-run zppy. Newly added task will be submitted, while previously completed ones will be skipped.

Tasks

If you run ls you’ll probably see a file like e3sm_diags_180x360_aave_model_vs_obs_0001-0020.status. This is one e3sm_diags job. Parts of the file name are explained below:

...

• <e3sm_simulations_dir>/<case_name>/post/atm/180x360_aave/ts/monthly/10yr has the time series files – one variable per file, in 10 year chunks as defined in <post<run_processingscripts_script_dir>/post.<case_name>.cfg.

• <e3sm_simulations_dir>/<case_name>/post/atm/180x360_aave/clim/20yr similarly has climatology files for 20 year periods, as defined in <post<run_processing_scriptscripts_dir>/post.<case_name>.cfg`.

• <e3sm_simulations_dir>/<case_name>/post/atm/glb/ts/monthly/10yr has globally averaged files for 10 years periods as defined in <post<run_processing_scriptscripts_dir>/post.<case_name>.cfg. The glb directory currently doesn't follow the same file naming convention as 180x360_aave.

Documenting the Model Run

...

Run the following commands:

cd <post<code_processingsource_script_dir>/zppy/global_time_series
# Edit the variables at the top of `generate_global_time_series_plots.sh`
# See below for an explanation of the variables
./generate_global_time_series_plots.sh

...

# For unified environment paths, see https://e3sm-project.github.io/e3sm_diags/docs/html/quickguides/quick-guide-general.html
unified_script=<the path to the unified environment script. Do NOT include `source`>                                                               
e3sm_simulations_dir=<simulations_dir>
case_dir=${e3sm_simulations_dir}/<case_name>
# For web directory paths, see https://e3sm-project.github.io/e3sm_diags/docs/html/quickguides/quick-guide-general.html
web_dir=<html_path>/E3SM/v2/beta/<case_name>
zppy_dir=<post<code_processingsource_script_dir>/zppy/

# Names                                                                  
moc_file=<e.g., mocTimeSeries_0001-0100.nc>
experiment_name=<e.g., 20210122.v2_test01.piControl.chrysalis>
figstr=<e.g., coupled_v2_test01>

# Years                                                                  
start_yr=<first year to process>
end_yr=<last year to process>

...

Your diagnostics are located at the web address corresponding to the www path in <post<run_processing_scriptscripts_dir>/<case_name>.cfg.

See the “Global time series” section above for finding the relevant web links. Fill the table with the specific web links: e.g., https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/<username>/E3SM/v2/beta/<case_name>/e3sm_diags/180x360_aave/model_vs_obs_0001-0020/viewer/.

...