...
ncclimo will reshape input files that are a series of snapshots of all model variables into outputs that are continuous timeseries of each individual variable taken from all input files. Timeseries to be reshaped (split) often come with hard-to-predict names, e.g., because the number of days or months in a file, or timesteps per day or month may all vary. Thus ncclimo in splitter mode requires the user to supply the input filenames. ncclimo will not construct input filenames itself in splitter mode (unlike monthly or annual climo generation mode). ncclimo will , as of version 5.0.4, employ employ timeseries reshaping mode if it receives the --split switch (recommended for clarity) or the --ypf_max option described below. In addition, it must receive a list of files through a pipe to stdin, or, alternatively, placed as positional arguments (after the last command-line option), or if neither of these is done and caseid is specified and the temporal resolution is monthly, it will automatically generate the filenames (identically to climatology mode), or if none of the previous four is done and no caseid caseid is specified, in which case it assumes it will assume that all *.nc files in drc_in constitute the input file list. These examples invoke reshaping mode in the four five possible ways (choose your poison):
# Sample Abbreviationsdrc_in=~zender/data/ne30/rawmap_fl=${DATA}/maps/map_ne30/map_ne30pg2_to_cmip6_180x360_nco.20200901.nc# Splitter Input Mode #1: Read input filename list from filels $drc_in/eamelm.h0.201[34].nc > input_listncclimo -P elm --split --yr_srt=2013 --yr_end=2014 --var=FSNTTBOT,AODVIS FNIR --map=$map_fl --drc_out=$drc_out < input_list# Splitter Input Mode #2: Pipe input filenames to stdincd $drc_inls $drc_in/eam.h0.201[34].nc | ncclimo -P eam --split --yr_srt=2013 --yr_end=2014 --var=FSNT,AODVIS --map=$map_fl --drc_out=$drc_out# Splitter Input Mode #3: Append filenames positional argumentsncclimo -P eam --split --var=FSNT,AODVIS --yr_srt=2013 --yr_end=2014 --map=$map_fl --drc_out=$drc_out $drc_in/cameam.h0.0[012]??.nc# Splitter Input Mode #4: Ingest entire directory (be sure the directory contains only files to be climatologized!)Automatically generate monthly input filenamesncclimo -P mpaso --split --var=timeMonthly_avg_activeTracers_temperature --yr_srt=2013 --yr_end=2014 --drc_in=$drc_in --map=$map_fl --drc_out=$drc_out
Splitter Input Mode #5: Ingest entire directory (be sure the directory contains only files to be split!)ncclimo -P eam --split --var=T,Q,RH --yr_srt=2013 --yr_end=2014 --drc_in=$drc_in --map=$map_fl --drc_out=$drc_out
...
The atmosphere and ocean model output is significantly larger than the land and ice model output. These commands recognize that by using different parallelization strategies that may be required, depending on the RAM fatness of the analysis nodes, as explained below. MPAS models do not utilize the $caseid option. They use their own naming convention. By default, ncclimo processes the MPAS hist.am.timeSeriesStatsMonthly analysis members.
Extended climos
...
ncclimo can re-use previous work and produce extended (i.e., longer duration) climatologies by combining two previously computed climatologies (this is called the binary method, and was introduced in NCO 4.6.3-alpha02) ), or by computing a new climatology from raw monthly model output and then combining that with a previously computed climatology (this is called the incremental method, and was introduced in NCO 4.6.2). ). Producing an extended climatology by the incremental method requires specifying (with -S and -s, respectively) the start years of the previously computed and current climo, and (with -e) the end year of the current climo. Producing an extended climatology by the binary method requires specifying both the start years (with -S and -s) and end years (with -E and -e) of both pre-computed climatologies. The presence of the -E option signifies to ncclimo to employ the binary (not incremental) method.
Following are two examples of computing extended climatologies using the binary method (i.e., both input climatologies are already computed using the normal methods above). If both input climatologies are in the same directory in which the output (extended) climatology is to be stored, then the number of required options is minimal:
...
caseid=somelongname
...
drc_in=/scratch1/scratchdirs/zender/e3sm/${caseid}/atm
...
ncclimo
...
-P eam -c
...
${caseid}
...
-
...
S
...
10
...
-E
...
20
...
-s
...
21
...
-e
...
50
...
-i
...
${drc_in}
When no output directory is specified, ncclimo’s internal logic automatically places the extended climo in the input climo directory. Files are not overwritten because the extended climos have different names filenames than the input climos. The next example utilizes the directory structure and options that Chris Golaz adopted for coupled ACME/of coupled E3SM simulations. The extra options (compared to the idealized example above) supply important information. The input climos were generated in seasonally discontiguous December (sdd) mode so the extended climatology must also be generated with the '-a sdd' option (or else ncclimo will not find the pre-computed input files). The input directory for the first pre-computed input climatology is specified with -x. The second pre-computed input climatology is specified with the usual -i option. A new output directory for the extended climos is specified with -X.
...
(NB: all of these single letter options have long-option name synonyms as described above.)
caseid=20161117.beta0.A_WCYCL1850S.ne30_oEC_ICG.edison
...
drc_ntv=/scratch2/scratchdirs/golaz/ACME_simulations/20161117.beta0.A_WCYCL1850S.ne30_oEC_ICG.edison/pp/clim
...
#
...
Native
...
drc_rgr=/scratch2/scratchdirs/golaz/ACME_simulations/20161117.beta0.A_WCYCL1850S.ne30_oEC_ICG.edison/pp/clim_rgr
...
#
...
Regridded
...
ncclimo
...
-P eam -a
...
sdd
...
-c
...
${caseid}
...
-
...
S
...
41
...
-E
...
50
...
-x
...
${drc_ntv}/0041-0050
...
-s
...
51
...
-e
...
60
...
-i
...
${drc_ntv}/0051-0060
...
-X
...
${drc_ntv}/0041-0060
...
ncclimo
...
-P eam -a
...
sdd
...
-c
...
${caseid}
...
-S
...
41
...
-E
...
50
...
-x
...
${drc_rgr}/0041-0050
...
-s
...
51
...
-e
...
60
...
-i
...
${drc_rgr}/0051-0060
...
-X
...
${drc_rgr}/0041-0060
The extended native and regridded climatologies are produced with virtually the same command (only the input and output directories differ). No mapping file or regridding option is necessary to produce an extended climatology from two input regridded climatologies. ncclimo need does not know or care whether the input climos are native-grid or are already regridded. So long as the regridded climatologies are already available, it makes more sense to re-use them rather than to perform a second regridding. While ncclimo can generate and regrid an extended climatology from native-grid inputs in one command, doing so involves more command-line options and it is generally simpler to follow the above procedure. Ask me if Charlie Zender if you would like help customizing ncclimo for other such workflows. Producing extended climatologies via the binary method consumes much less memory than producing normal or incremental climatologies. The binary method simply computes weighted averages of each input variable. Hence the maximum RAM required is approximately only three times the size of the largest input variable. This is trivial compared to the total input file size, hence the extended climos may be computed with background parallelism, the default in ncclimo. The '-p mpi' option is never necessary for producing extended climos using the binary method. As you might imagine, the combination of low memory overhead and re-use of previously regridded climos means that producing extended regridded climos via the binary method is extremely fast compared to computing normal climos. Binary climos (regridded or not) require only about 1 minute on Edison.
Memory Considerations
...
It is important to employ the optimalncclimo parallelization strategy for your computer hardware resources. Select from the three available choices with the '-p par_typ' switch option. The options are serial mode ('-p nil' or '-p serial'), background mode parallelism ('-p bck'), and MPI parallelism ('-p mpi'). The default is background mode parallelism, which is appropriate for lower resolution (e.g., ne30L72) simulations on most nodes at high-performance computer centers. Use (or at least start with) serial mode on personal laptops/workstations. Serial mode requires twelve times less RAM than the parallel modes, and is much less likely to deadlock or cause OOM (out-of-memory) conditions on your personal computer. If the available RAM (+swap) is < 12*4*sizeof(monthly input file), then try serial mode first (12 is the optimal number of parallel processes for monthly climos, the computational overhead is a factor of four). EAMv1 ne30L72 ne30np4L72 output is about ~1 GiB per month so each month requires about 4 GiB of RAM. EAMv1 ne30L72 ne30np4L72 output (with LINOZ) is about ~10 GB/month so each month requires ~40 GiB RAM. EAMv1 ne120L72 ne120np4L72 output is about ~12 GB/month so each month requires ~48 GB RAM. The computer does not actually use all this memory at one time, and many kernels compress RAM usage to below what top reports, so the actual physical usage is hard to pin-down, but may be a factor of 2.5-3.0 (rather than a factor of four) times the size of the input file. For instance, a 16 GB MacBookPro will successfully run an ne30L30 climatology (that requests 48 GB RAM) in background mode, but the laptop will be slow and unresponsive for other uses until it finishes (in 6-8 minutes) the climos. Experiment a bit and choose the parallelization option that works best for you.
Serial mode, as its name implies, uses one core at a time for climos, and computes sequentially the monthly then seasonal then annual climatologies. Serial mode means that climos are performed serially, but regridding will employ OMP threading on platforms that support it, and use up to 16 cores. By design each month and each season are independent of the others, so all months can be computed in parallel, then each season can be computed in parallel (using monthly climatologies), then the annual average can be computed. Background parallelization mode exploits this parallelism and executes the climos in parallel as background processes on a single node, so that twelve cores are simultaneously employed for monthly climatologies, four for seasonal, and one for annual. The optional regridding will employ up to two cores per process. MPI parallelism executes the climatologies on different nodes so that up to (optimally) twelve nodes are employed performing monthly climos. The full memory of each node is available for each individual climo. The optional regridding will employ up to eight cores per node. MPI mode or Background mode on a big memory queue must be used to process ne30L72 and ne120L72 climos on some, but not all, DOE computers. For example, attempting an ne120L72 ne120np4L72 climo in background mode on Cori (i.e., on one a 96 GB compute node ) will fail due to OOM. (OOM errors do not produce useful return codes so if your climo processes die without printing useful information, the cause may be OOM). However the same climo will succeed if executed on a single big-memory (1 TB) node on Cori (use -C amd on Cori, or -lpartition=gpu on Andes, as shown below). 512 GB node. Or MPI mode can be used for any climatology. The same ne120L72 ne120np4L72 climo will also finish blazingly fast in background mode on Cooley (i.e., on one 384 a 512 GB compute node), so MPI mode is unnecessary on the beefiest nodes. In general, the fatter the memory, the better the performance.
This implementation of parallelism for climatology generation once had relatively poor granularity. Meaning that nodes using background or parallel mode always computed 12 monthly climatologies simultaneously, and nodes using serial mode always computed only 1 climatology at a time, and there was no granularity in between these extremes. The '-j job$job_nbr' option (also in ncremap) allows the user to specify the exact granularity to match the node's resources. Here job $job_nbr specifies the maximum number of simultaneous climo tasks (averaging, regridding) to send to a node at one time. The default value of job_nbr is 12 for monthly climatologies in both MPI and Background parallelism modes. This can be over-ridden to improve granularity. For example, if --job_nbr=4 is explicitly requested, then the 12 monthly climos will be computed in three sequential batches of four months each. ncclimoautomatically sets job_nbr to the number of nodes available when working in splitter (not climo) mode, so invoking ncclimowith four nodes in splitter mode means each of the four nodes will receive one splitter task. In Background mode job_nbr defaults to 12 and if job_nbr is explicitly specified, say with --job_nbr=4, then four months are computed simultaneously on the host node. Some nodes, e.g., your personal workstation, are underpowered for 12 climo tasks yet overpowered for 1 task, and so benefit from improved granularity.
...
Climos on Single Compute Nodes at LCFs
The basic approach above (running the script from a standard terminal window) works well for small cases, yet can be unpleasantly slow on login nodes of LCFs and for longer or higher resolution (e.g., ne120) climatologies. As a baseline, generating a climatology of 5 years of ne30 ne30pg2 (~1x1 degree) EAM output with ncclimo takes 1-2 minutes on rhea (at a time with little contention), and 6-8 minutes on a 2014 MacBook Pro. To make things a bit faster at LCFs, you can ask for your own dedicated node (note this approach doesn't does not make sense except on supercomputers which that have a job-control queue). On rhea Perlmutter do this via:
qsub -I srun -A CLI115 e3sm -V -l nodes=1 -l walltime=00:10:00 -N ncclimo # rhea standard node (128 GB)
qsub -I -A CLI115 -V -l nodes=1 -l walltime=00:10:00 -lpartition=gpu -N ncclimo # rhea bigmem node (1 TB)
The equivalents on cooley and cori are:
qsub -I -A OceanClimate_4 --nodecount=1 --constraint=cpu --nodes=1 --time=00:1030:00 --jobname=ncclimo # cooley node (384 GB)salloc -A e3sm --nodes=1 --partition=debug --time=00:10:00 --job-name=ncclimo # cori node (128 GB)qos=debug --job-name=ncclimo --pty bash
Acquiring a dedicated node is useful for any calculation you want to do quickly, not just creating climo files though it does burn through our computing allocation so don't be wastefulprudent. This command returns a prompt once a nodes are is assigned (the prompt is returned in your home directory so you may then have to cd to the location you meant mean to run from). At that point you can simply use the 'basic' ncclimo invocation to run your codeinvoke ncclimo as described above. It will be faster because you are not sharing the node it's running on with other people. Again, ne30L30 ne30pg2L72 climos only require < 2 minutes, so the 10 30 minutes requested in the example is excessive and conservative. Tune-it with experience. Here is the meaning of each flag used:
-I (that's a capital "i”): submit in interactive mode = return a prompt rather than running a program.A: Name of the account to charge for time used--time: how long to keep this dedicated node for. Unless you kill the shell created by the qsub command, the shell will exist for this amount of time, then die suddenly. In the above examples, 3 hrs is requested.-l nodes=1 (rhea) or --nodecount 1 (cooley) or --nodes=1 (cori): the number constraint: Queue name--nodes=1: Number of nodes to request. ncclimo will use multiple cores per node.-V: export existing environmental variables into the new interactive shell. Peter doubts this is actually needed.
-q: the queue name (needed for locations like edison that have multiple queues with no default queue)
-A: the name of the account to charge for time used. This page may be useful for figuring that out if the above defaults don't work: /wiki/spaces/ED/pages/1114710
...
-time: How long to keep this dedicated node for--pty bash: Submit in interactive mode = return a prompt rather than running a program
Climos on Multiple Nodes at LCFs
The above parallel approaches will fail when a single node lacks enough RAM (plus swap) to store all twelve monthly input files, plus extra RAM for computations. One should employ MPI multinode parallelism (-p mpi) on nodes with less RAM than 12*3*sizeof(monthly input). The longest an ne120 climo will take is less than half an hour (~25 minutes on Edison or Rhea), so the simplest method to run MPI jobs is to request 12-interactive nodes using the above commands (though remember to add -p mpi), then execute the script at the command line. It is also possible, and sometimes preferable, to request non-interactive compute nodes in a batch queue. Executing an MPI-mode climo (on machines with job scheduling and, optimally, 12 available nodes) in a batch queue can be done in 2 commands. First, write an executable file which calls the ncclimo script with appropriate arguments. We do this below by echoing to a file ~/ncclimo.pbs, but you could also open an editor and copy the stuff in quotes below into a file and save it:
...