...
You can get out of memory errors in PIO if the number of tasks is too small. But if the number of tasks is too large, we get out of memory errors in the MPI library itself because there are too many messages. At the same time the I/O performance is better for large data sets with more io I/O tasks.
Users have to typically start with the default configuration and try couple of PIO_NUMTASKS and PIO_STRIDE options to get an optimal PIO layout for a model configuration on a particular machine. On large machines, empirically we have found that we need to constrain the number of PIO io I/O tasks (PIO_NUMTASKS) to less than 128 to prevent out of memory errors.
...