...
To understand the impact of I/O on the model performance start by taking a look at the time taken by the PIO functions (grep for pio*) in the timing subdirectory of the job run directory. (At NERSC, ALCF, and OLCF, these data are also archived in the performance_archive for each system. Site is system specific.) Bad PIO settings can cause the model to crash, or have horribly slow output performance. But you want to make sure you have a reasonable processor layout before calibrating the PIO settings.
(At NERSC, ALCF, and OLCF, the timing and other performance-related data are also archived in the performance_archive for each system. Site is system specific. Currently:
- NERSC: /project/projectdirs/(project id)/performance_archive
- ALCF: /projects/(project id)/performance_archive
- OLCF: /lustre/atlas/proj-shared/(project id)/performance_archive
)
Several aspects of the model I/O performance can be controlled at runtime by setting the PIO* parameters in env_run.xml. The model developer can start by tuning the number of PIO I/O tasks (processes that perform I/O) and the stride between the I/O tasks by setting PIO_NUMTASKS and PIO_STRIDE respectively in env_run.xml. By default the PIO_STRIDE is set to 4, which means as you increase the number of tasks, you get more tasks trying to write to the filesystem.
...