CDF (Common Data Format) is a file format frequently used to store multidimensional data in a self describing data format.
In E3SM we currently store all input and output files in the CDF format. These input and output files are processed by PIO using NetCDF and PnetCDF libraries. Moreover, some of the input files are generated by other third party libraries/software packages that support the CDF file format
Since different libraries implement the CDF file format, bugs/issues in the implementation can create/process files that are not CDF compatible or valid. So for example, an invalid CDF file that can be processed (read) by NetCDF may not work with PnetCDF. Please use the steps below to test for the validity of a CDF file (When creating input files please use the steps below to ensure that PIO can process the input files).
The PnetCDF library provides a utility, ncvalidator, to check if a file is a valid CDF file. Use the ncvalidator tool to check if a file is a valid CDF file. The tool prints out detailed information on any compatibility issues with CDF. On Anvil (Blues login node), we have PnetCDF 1.9.0 installed at /lcrc/group/acme/soft/pnetcdf/1.9.0/intel-17.0.0/mvapich2-2.2 and the ncvalidator tool is available at /lcrc/group/acme/soft/pnetcdf/1.9.0/intel-17.0.0/mvapich2-2.2/bin/ncvalidator.
[jayesh@blogin4 ~]$ /lcrc/group/acme/soft/pnetcdf/1.9.0/intel-17.0.0/mvapich2-2.2/bin/ncvalidator --help /lcrc/group/acme/soft/pnetcdf/1.9.0/intel-17.0.0/mvapich2-2.2/bin/ncvalidator: invalid option -- '-' Usage: /lcrc/group/acme/soft/pnetcdf/1.9.0/intel-17.0.0/mvapich2-2.2/bin/ncvalidator [-h] | [-q] file_name [-h] Print help [-x] repair null-byte padding in file header [-q] Quiet mode (exit 1 when fail, 0 success) filename: input netCDF file name PnetCDF library version 1.9.0- The newer versions of PnetCDF provide an option, -x, to ncvalidator to correct a frequently occurring CDF compatibility issue (null byte padding in the header). Use this option to see if you can resolve the CDF compatibility issue. Note that using this option will modify the input file.
/lcrc/group/acme/soft/pnetcdf/1.9.0/intel-17.0.0/mvapich2-2.2/bin/ncvalidator -x myinputfile.nc
- There is a simple PnetCDF program that tries to open/close a file using PnetCDF available on Anvil at /lcrc/group/acme/soft/pnetcdf/1.6.1/intel-17.0.0/mvapich2-2.2/test/test_open_close . This test program can be used to check if a file can be opened/closed by PnetCDF. We plan to add similar tools to PIO2 to make these tests easier in future. However note that different versions of PnetCDF have different level of "strictness" wrt to adherence to the CDF standard (the invalid CDF file in the example below can be successfully opened/closed with PnetCDF 1.9.0 and multiple versions of NetCDF, however the file cannot be opened with PnetCDF 1.6.1)
[jayesh@blogin4 ~]$ /lcrc/group/acme/soft/pnetcdf/1.6.1/intel-17.0.0/mvapich2-2.2/test/test_open_close Usage : /lcrc/group/acme/soft/pnetcdf/1.6.1/intel-17.0.0/mvapich2-2.2/test/test_open_close [file name] // A valid CDF file - can be opened/closed by PnetCDF [jayesh@blogin4 test]$ /lcrc/group/acme/soft/pnetcdf/1.6.1/intel-17.0.0/mvapich2-2.2/test/test_open_close CNP_parameters_c180524.nc [0/1] Trying to open file : CNP_parameters_c180524.nc [0/1] Successfully opened CNP_parameters_c180524.nc, trying to close the file now [0/1] Successfully closed CNP_parameters_c180524.nc // An invalid CDF file - cannot be opened by PnetCDF [jayesh@blogin4 test]$ /lcrc/group/acme/soft/pnetcdf/1.6.1/intel-17.0.0/mvapich2-2.2/test/test_open_close orig_CNP_parameters_c180312.nc [0/1] Trying to open file : orig_CNP_parameters_c180312.nc [0/1] Opening file (orig_CNP_parameters_c180312.nc) failed : NetCDF: Invalid argument