How to run RRTMGP standalone test

Owned by Robert Jacob
Oct 17, 2019

(Written by Ben Hillman over in the ECP Confluence.  Copied here).

The RRTMGP radiative transfer code is maintained in an external repository, and can be run as a standalone code for example problems. This is useful both for regression testing of the code itself, and for performance benchmarks. The bulk of the steps out outlined in the external RRTMGP repo (https://github.com/RobertPincus/rte-rrtmgp), but instructions specific to running on Summit follow below.

Step-by-step guide


Obtain copy of external RRTMGP code:

The code exists in a git repository maintained by Robert Pincus here: https://github.com/RobertPincus/rte-rrtmgp. Clone into directory of your choosing:

git clone git@github.com:RobertPincus/rte-rrtmgp.git rte-rrtmgp

Build the rte-rrtmgp library

  1. First, change directory to the build directory within the repo just cloned: `cd rte-rrtmgp/build`

  2. Make symbolic link to OLCF PGI version of Makefile.conf: `ln -s Makefile.conf.olcf_pgi Makefile.conf`

  3. For the GPU, set the environment variable RTE_KERNELS=openacc to build the OpenACC version of the kernels : `export RTE_KERNELS=openacc`
  4. For the CPU-only version, leave RTE_KERNELS unset
  5. Make sure the pgi module is loaded to use the PGI compiler: `module load pgi`
  6. And finally, build the code: `make`


Build and run RFMIP example

  1. Included with the RRTMGP repo is code to run an example problem, the RFMIP clearsky case. First, we need to build the code to run these (it's a separate driver). Change directory to the location of the code: `cd ../examples/rfmip-clear-sky`
  2. Make a symbolic link to OLCF Makefile.libs: `ln -s Makefile.libs.olcf Makefile.libs`
  3. Set environment variable RRTMGP_DIR to directory where the RTE-RRTMGP library was built in the previous step: `export RRTMGP_DIR=../../build`
  4. Load netcdf modules: `module load netcdf netcdf-fortran`
  5. Build code: `make`
  6. We now need to obtain the inputdata to run the example. There are some scripts to do this, but we need a custom python environment with some extra libraries to do so. In particular, netCDF4 and urllib. Files can then be downloaded by running the script "stage_files.py". This will put a few netCDF files in the current directory. We will also need the absorption coefficient input files from the RRTMGP directory (in rrtmgp/data in the root of the repository). Because we will need to run somewhere on /gpfs on summit in order to run on compute nodes, we just copy the inputfiles and executables to a new directory in $PROJWORK, and "cd" there to run:
rundir=${PROJWORK}/cli115/${USER}/rte-rrtmgp/cases/rfmip-clear-sky
mkdir -p ${rundir}

# Copy data and executables
cp -v *.nc ${rundir}/
cp -v ../../rrtmgp/data/*.nc ${rundir}/
cp -v rrtmgp_rfmip_sw ${rundir}/
cp -v rrtmgp_rfmip_lw ${rundir}/

# Run
cd ${rundir}
rfmip_file=multiple_input4MIPs_radiation_RFMIP_UColorado-RFMIP-1-2_none.nc
kdist_sw=rrtmgp-data-sw-g224-2018-12-04.nc
kdist_lw=rrtmgp-data-lw-g256-2018-12-04.nc
echo "jsrun -n 1 -c 1 -g 1 ./rrtmgp_rfmip_lw 1800 ${rfmip_file} ${kdist_lw}" > runscript_lw.sh
echo "jsrun -n 1 -c 1 -g 1 ./rrtmgp_rfmip_sw 1800 ${rfmip_file} ${kdist_sw}" > runscript_sw.sh
bsub -nnodes 1 -P cli115 -W 00:01 -I runscript_lw.sh
bsub -nnodes 1 -P cli115 -W 00:01 -I runscript_sw.sh