Testing for NetCDF Compatibility/Validity

Owned by Jayesh Krishna
Last updated: May 29, 2018
2 min read

NetCDF (Network Common Data Form) is a file format frequently used to store multidimensional data in a self describing data format.

In E3SM we currently store all input and output files in the NetCDF format. These input and output files are processed by PIO using NetCDF and PnetCDF libraries. Moreover, some of the input files are generated by other third party libraries/software packages that support the NetCDF file format

Since different libraries implement the NetCDF file format, bugs/issues in the implementation can create/process files that are not NetCDF compatible or valid. So for example, an invalid NetCDF file that can be processed (read) by the NetCDF library may not work with PnetCDF. Please use the steps below to test for the validity of a NetCDF file (When creating input files please use the steps below to ensure that PIO can process the input files).

  • The PnetCDF library provides a utility, ncvalidator, to check if a file is a valid CDF file. Use the ncvalidator tool to check if a file is a valid CDF file. The tool prints out detailed information on any compatibility issues with NetCDF. On Anvil (Blues login node), we have PnetCDF 1.9.0 installed at /lcrc/group/acme/soft/pnetcdf/1.9.0/intel-17.0.0/mvapich2-2.2 and the ncvalidator tool is available at /lcrc/group/acme/soft/pnetcdf/1.9.0/intel-17.0.0/mvapich2-2.2/bin/ncvalidator.

    [jayesh@blogin4 ~]$ /lcrc/group/acme/soft/pnetcdf/1.9.0/intel-17.0.0/mvapich2-2.2/bin/ncvalidator --help
/lcrc/group/acme/soft/pnetcdf/1.9.0/intel-17.0.0/mvapich2-2.2/bin/ncvalidator: invalid option -- '-'
Usage: /lcrc/group/acme/soft/pnetcdf/1.9.0/intel-17.0.0/mvapich2-2.2/bin/ncvalidator [-h] | [-q] file_name
       [-h] Print help
       [-x] repair null-byte padding in file header
       [-q] Quiet mode (exit 1 when fail, 0 success)
       filename: input netCDF file name
       PnetCDF library version 1.9.0
  • The newer versions of PnetCDF provide an option, -x, to ncvalidator to correct a frequently occurring NetCDF compatibility issue (null byte padding in the header). Use this option to see if you can resolve the NetCDF compatibility issue. Note that using this option will modify the input file.
    /lcrc/group/acme/soft/pnetcdf/1.9.0/intel-17.0.0/mvapich2-2.2/bin/ncvalidator -x myinputfile.nc
  • There is a simple PnetCDF program that tries to open/close a file using PnetCDF available on Anvil at /lcrc/group/acme/soft/pnetcdf/1.6.1/intel-17.0.0/mvapich2-2.2/test/test_open_close . This test program can be used to check if a file can be opened/closed by PnetCDF. We plan to add similar tools to PIO2 to make these tests easier in future. Note that different versions of PnetCDF have different level of "strictness" wrt to adherence to the NetCDF standard (the invalid CDF file in the example below can be successfully opened/closed with PnetCDF 1.9.0 and multiple versions of NetCDF, however the file cannot be opened with PnetCDF 1.6.1)
    [jayesh@blogin4 ~]$ /lcrc/group/acme/soft/pnetcdf/1.6.1/intel-17.0.0/mvapich2-2.2/test/test_open_close
Usage : /lcrc/group/acme/soft/pnetcdf/1.6.1/intel-17.0.0/mvapich2-2.2/test/test_open_close [file name]
// A valid CDF file - can be opened/closed by PnetCDF
[jayesh@blogin4 test]$ /lcrc/group/acme/soft/pnetcdf/1.6.1/intel-17.0.0/mvapich2-2.2/test/test_open_close CNP_parameters_c180524.nc 
[0/1] Trying to open file : CNP_parameters_c180524.nc
[0/1] Successfully opened CNP_parameters_c180524.nc, trying to close the file now
[0/1] Successfully closed CNP_parameters_c180524.nc
// An invalid CDF file - cannot be opened by PnetCDF
[jayesh@blogin4 test]$ /lcrc/group/acme/soft/pnetcdf/1.6.1/intel-17.0.0/mvapich2-2.2/test/test_open_close orig_CNP_parameters_c180312.nc 
[0/1] Trying to open file : orig_CNP_parameters_c180312.nc
[0/1] Opening file (orig_CNP_parameters_c180312.nc) failed : NetCDF: Invalid argument