| Warning | ||
|---|---|---|
| ||
NERSC Directory Change Notice Due to project's name change at NERSC from 'ACME' to 'E3SM' and NERSC’s file system update, the directory ' |
Table of Contents
| Table of Contents |
|---|
Child Pages
Child pages (Children Display)
The purpose of this page is to document the procedure for adding support for new atmosphere grids. The process should be the same for new uniform resolutions as well as for new regionally-refined meshes, although some settings will need to be changed for new regionally-refined mesh configurations. This page is a work in progress, and will be updated as this process is refined and (eventually) made more automated. This documentation is an update of a document written by Mark Taylor and Colin Zarzycki, available as a Google Doc here. Linking to a related page: /wiki/spaces/NGDNA/pages/915408074.
For the purpose of this step-by-step guide, we will walk through each step using an example of regenerating the ne4 grid that is already currently supported. For a new grid (i.e., ne512), just replace ne4 with the desired grid identifier. This is determined in Step 1 Generate a new "grid" file below, and the rest should follow.
For this example, I will assume a few environment variables, which we should set now. First, determine where to put the generated output files. For big grids, this can take up quite a bit of space, so choose a location with sufficient space. I have a symlink to my CSCRATCH directory on NESRC in my home directory, so I will put grid files there for this example:
| Code Block |
|---|
output_root=${HOME}/cscratch/e3sm/grids/ne4
mkdir -p ${output_root} |
Types of Atmosphere grid metadata files
See CAM-SE Grid Overview for description of the spectral elements, GLL nodes, subcell grid and dual grid.
- Exodus file: "ne4.g". This is a netcdf file following Exodus conventions. It gives the corners of all elements on the sphere and their connectivity. It is independent of the polynomial order used inside the element ("np").
- This file is used by TempestRemap (TR) to generate mapping files. The polynomial order is a command line option and the GLL nodes are internally generated by TR.
- latlon file: "ne4np4_latlon.nc". This file contains a list of all the GLL nodes in the mesh (in latitude/longitude coordinates). The list of GLL nodes must be in the the internal HOMME global id ordering, matching the ordering used in CAM and EAM native grid output. It also contains the connectivity of the GLL subcell grid.
- This file is used by CAM's interpic_new utility, and graphics programs Paraview and Visit when plotting native grid output.
- SCRIP file: "ne4np4_scrip.nc". This file contains a description of the GLL dual grid, in the format used by the original incremental remap tool SCRIP. It includes the locations of the GLL nodes and artificial bounding polygons around those nodes. Ideally the spherical area of each polygon will match the GLL weight ("exact GLL areas"), but not all tools can achieve exact areas. Inexact areas does not impact the accuracy of the resulting mapping algorithm, it just means that mass will not be exactly conserved by the mapping algorithm.
- Needed in E3SMv2 configurations by the cube_to_target topography dataset generation tool.
- Historically used by CESM/E3SMv1 to create ocean/atm mapping files (exact GLL areas are important here, to conserve fluxes)
- Used by the regional and sub-gridscale regridding post-processing tools ( https://acme-climate.atlassian.net/wiki/spaces/SIM/pages/754286611/Regridding+E3SM+Data+with+ncremap, http://nco.sf.net/nco.html#sgs)
Step-by-step guide
1. Generate a new "grid" file
Requirements: TempestRemap
In order to generate mapping files to do interpolation between a new atmosphere grid and the other component grids, we need to create a file that describes the atmosphere grid representation. The TempestRemap tool (https://github.com/ClimateGlobalChange/tempestremap) can be used to recreate the internal representation of the spectral element grids used in HOMME. The grid mesh file will be saved in an "exodus" type file (with a .g extension).
TempestRemap needs to be built from source from the Github repository. This is straightforward on Cori and Edison. Note that a parallel version of netCDF should be used. This can be accomplished on cori/edison be executing module swap cray-netcdf cray-netcdf-hdf5parallel or by sourcing an env_mach_specific.sh from a working case on the target machine before building. Then,
| Code Block |
|---|
git clone https://github.com/ClimateGlobalChange/tempestremap.git
cd tempestremap
make -f Makefile.gmake
cd .. |
This will build TempestRemap into tempestremap/bin. For the rest of the tutorial, assume that the environment variable ${tempest_root} points to this directory:
| Code Block |
|---|
tempest_root=${PWD}/tempestremap |
We can generate an ne4 mesh easily now:
| Code Block |
|---|
${tempest_root}/bin/GenerateCSMesh --alt --res 4 --file ${output_root}/ne4.g |
For RRM grids, SQuadGen should be used (available in a separate repository on Github here). This will be documented in another page (TODO).
The Exodus file contains only information about the position of the spectral element on the sphere. For SE aware utilities such as TempestRemap, they can use the polynomial order and the reference element map to fill in necessary data such as the locations of the nodal GLL points. For non-SE aware utilities, we need additional meta data, described in the next section.
2A. Generate "dual grid" mesh files for E3SM v2 "pg2" grids
Requirements:
- exodus mesh file
- TempestRemap
In E3SM v2, we will be switching to running physics on a FV pg2 grid and mapping files between the atmosphere and other components will be FV to FV type maps using the pg2 grids. These can be generated by TempestRemap directly from the exodus mesh file:
| Code Block |
|---|
${tempest_root}/bin/GenerateVolumetricMesh --in ne4.g --out ne4pg2.g --np 2 --uniform
${tempest_root}/bin/ConvertExodusToSCRIP --in ne4pg2.g --out ne4pg2.scrip.nc |
2B. Generate "dual grid" mesh files (SCRIP and lat/lon format) for E3SM v1 "np4" GLL grids
Requirements:
- exodus mesh file
- Matlab or Fortran utility.
Note: in E3SM v2, we will be switching to running physics on a FV pg2 grid and mapping files between the atmosphere and other components will be FV to FV type maps using the pg2 grids. The spectral element "np4" grid is still used internally by the dynamics and for initial conditions, so the metadata described in this section is still needed for some analysis and preparation of initial conditions.
Mapping files used by the coupler for mapping fluxes between SE "np4" and FV grids should be generated with TempestRemap and only need the exodus grid description (which provides the locations of the corners of the quadrilaterals that form the elements of the cube-sphere mesh) generated in Step 1 above. However, a handful of pre- and post-processing tools require a finite-volume equivalent of the spectral element grid (these tools include the surface roughness calculation in the land tool cube_to_target, ESMF mapping files used for interpolating land surface data to target grid, and post-processing regional and subgrid remapping tools). We refer to this finite-volume description of the SE grid as the "dual grid" to the SE grid (see page describing atmosphere grids in more detail here).
The dual grid is generated using separate tools that draw polygons around the GLL nodes, and optionally optimize the area contained within each polygon to match the GLL weights. The resulting grid is saved in netCDF files that contain the grid point locations, area weights, and areas associated with each nodal value. In the SE dycore, this data depends on the reference element map (how the finite elements are mapped into the unit square reference element). The SE dycore supports two such reference element maps: the older "gnomonic equal angle" map, and an "element-local" map. RRM grids are required to use the "element-local" map. Cubed-sphere grids can use either map (chosen via namelist option), with the existing compsets in E3SM v1 all using the gnomonic equal area map. For E3SM v2, all new grids (cubed-sphere and RRM) should use the element-local map.
There are three separate codes that can create a dual grid from our SE grids:
- Matlab code. This code produces the best SCRIP files, with nice (non-starlike) control volumes around each GLL node, with exact areas (spherical area = GLL weight). But the serial code is painfully slow and thus is only practical for grids with up to 100,000 elements. This code only supports the element-local map. Noel Keen profiled the matlab script and found that at least for ne30 case, ~90% of the time is spent in computing the area of a polygon which is called millions of times. There may be some simple things to try to improve outside of rewriting.
- Fortran code. This utility is included with standalone HOMME. It runs quickly, in parallel, and produces exact GLL areas for cubed-sphere meshes. It supports both element maps via namelist options. But for RRM meshes it produces suboptimal control volumes (including star shaped control volumes) with only approximate areas (spherical area ≠ GLL weight). The resulting RRM SCRIP file is thus not acceptable for generating mapping files for coupled simulations, but they should be fine for generating surface roughness fields.
- NCAR utility. Will update when we find out more about this.
For the majority of use cases, optimizing the area and shape of each polygon in the dual grid is probably not necessarily, and a simple (and fast) approach can be taken using the Fortran code (plus some NCL scripts), which is packaged with HOMME. For conservative remapping, optimal areas that exactly match the GLL weights probably is required, and the slower Matlab code will most likely need to be used (NOTE: we plan to rewrite the Matlab code in a compiled language to work with higher resolution grids, but for the time being we will work with the less accurate Fortran/NCL codes for the few use cases we still need the SCRIP files for).
The _latlon file is a separate grid descriptor file that contains just a simple list of coordinates of all GLL nodes and is easy to produce. Either the Fortran+NCL or the Matlab utilities can produce this file.
The Fortran Code
To run the fortran code on a E3SM supported platform: (updated 2020/5/25 with updated tool from https://github.com/E3SM-Project/E3SM/pull/3593 )
- Be sure your environment matches the software environment loaded by E3SM by executing the output of this command: e3sm/cime/scripts/Tools/get_case_env
- Use cmake to configure and compile standalone HOMME. On a supported platform with the CIME environement, this should work out-of-the-box. See e3sm/cime/components/homme/README.cmake
- compile the HOMME tool utility:
- cd /path/to/workingdir
- make -j4 homme_tool
- executable: /path/to/workingdir/src/tool/homme_tool
- Edit e3sm/components/homme/test/tool/namelist/template.nl and specify the grid resolution or RRM file
- For ne512, this would be
set ne = 512. For RRM grids, leavene = 0, but will need to edit where the exodus grid file comes from - for non-RRM grids using the older E3SM v1 dycore, add cubed_sphere_map=0 to template.nl
- For ne512, this would be
- See e3sm/components/homme/test/tool/test.job for examples of how to run the script and then use an NCL utilities to process the tool output into SCRIP and latlon formats.
Specific details for running at NERSC on Cori(knl):
- Create a batch script hange "account" in the sbatch directives at the top of the script. For example, set
#SBATCH --account=e3sm - cmake -C /path/to/e3sm/components/homme/cmake/machineFiles/cori-knl.cmake -DPREQX_NP=4 /path/to/workingdir
- Make sure a working NCL is in your PATH. On Cori, add the following to the script:
module load ncl.
With these changes, run makegrid.job TWICE to build and then run the code. The first pass builds HOMME, and the second pass runs HOMME and the NCL utilities. In a HPC environment, the second pass needs to submitted to the batch queue via sbatch -C knl makegrid.job. This will create the grid descriptor files in ${HOME}/scratch1/preqx/template.
The Matlab Code
The tool chain described below only supports the element-local map. As noted above, this method produces the highest-quality output but is extremely slow (days to weeks) to run. Our preferred mapping algorithms (Transition to TempestRemap for Atmosphere grids) also only support the element-local map.
Creating the grid descriptor files for RRM grids and cubed-sphere grids using the element local map is accomplished using a matlab code written for this process, available in the PreAndPostProcessingScripts repo. First, checkout the code:
| Code Block |
|---|
# Checkout code
git clone git@github.com:ACME-Climate/PreAndPostProcessingScripts.git
# Set variable so we can refer to this directly below
PreAndPostProcessingScripts=${PWD}/PreAndPostProcessingScripts |
Now we can run the code by executing the run_dualgrid.sh script from the ${PreAndPostProcessingScripts}/regridding/spectral_elements_grid_utilities/compute_dualgrid directory. Note that a working Matlab environment is required to run the code. On NERSC, this can be accomplished by simply executing module load matlab before running the run_dualgrid.sh script.
| Code Block |
|---|
grid_name=ne4
mesh_file=${output_root}/ne4.g
module load matlab
cd ${PreAndPostProcessingScripts}/regridding/spectral_elements_grid_utilities/compute_dualgrid
./run_dualgrid.sh ${grid_name} ${mesh_file} |
This will create two files in the compute_dualgrid directory: a "latlon" and a "scrip" grid descriptor file. Move these to wherever you want to keep your grid files:
| Code Block |
|---|
mv *_latlon.nc ${output_root}
mv *_scrip.nc ${output_root} |
3. Generate mapping files
Requirements:
- TempestRemap
- ESMF_RegridWeightGen
- ncremap
- grid descriptor files for each component that exists on a different grid
(atmosphere, ocean, possibly land if on a different grid than the atmosphere)
In order to pass data between different components at runtime, a set of mapping files between each component is generated offline. These mapping files will also be used in Step 4 below (generating domain files).
For "pg2" grids used in E3SM v2, the recommended mapping files are slightly different and still a work in progress. See Transition to TempestRemap for Atmosphere grids
TempestRemap and ESMF are the backends that generate the mapping weights, but this is all nicely encapsulated using ncremap. Tempest is the preferred method for creating mapping files. The ncremap function, calling TempestRemap or ESMF, will decide which of these two tools to use based on the atmospheric input file. If the *.scrip file was used, then ESMF will be called. If the *.g file was used, then TempestRemap will be called. The ESMF tools are adequate for making atmosphere-only-type component sets for E3SM, but this tool is less conservative than TempestRemap. If you are making grids for a coupled run, then TempestRemap should be used.
The easiest way to make sure ncremap is up to date and will work for this workflow, we can source the ESMF-Unified conda environment. Instructions for supported machines can be found here. For Edison and Cori, it is as simple as:
| Code Block |
|---|
source /global/project/projectdirs/acme/software/anaconda_envs/load_latest_e3sm_unified.sh |
For unsupported machines you may need to install the conda environment via:
| Code Block |
|---|
conda install -c conda-forge -c e3sm -c cdat e3sm-unified |
TempestRemap was built in Step #1 above. We need to tell ncremap where to find our TempestRemap binaries, so we need to append to our PATH:
| Code Block |
|---|
PATH=${tempest_path}/bin:${PATH} |
And now we can use ncremap to generate ALL the needed mapping files between two grids, in this example the ne4 atmosphere and the oQU240 ocean grid (for the moment, we will put the land on the same grid as the atmosphere):
| Code Block |
|---|
atm_grid_file=ne4.g
ocn_grid_file=/project/projectdirs/acme/inputdata/cpl/gridmaps/oQU240/ocean.QU.240km.scrip.151209.nc
cd ${output_root} && ncremap -P mwf -s $ocn_grid_file -g $atm_grid_file --nm_src=oQU240 --nm_dst=ne4np4 --dt_sng=20181114 |
In the above code, the "-P mwf" option triggers a specific procedure that invokes multiple ncremap commands which each invoke multiple Tempest commands.It produces 3 mapping files each for a2o and o2a. This multitude of mapping files for each component→component case are needed because fluxes need conservative mapping and states need non-conservative mapping. Because "-P mwf" results in a great deal of output being flushed to screen, it is mostly suppressed by default. To see all the output or to figure out how to run all the nco and tempest remap commands one-by-one, add "--dbg_lvl=2" to the ncremap command. This will print the commands but not execute them. The user can then use one of the printed commands, again with "--dbg_lvl=2" to see the actual commands being sent to Tempest.
If this command is successful, it should produce many mapping files in ${output_root}, that look something like
| Code Block |
|---|
map_${atm_grid}_to_${ocn_grid}_${method}.${datestring}.nc
map_${ocn_grid}_to_${atm_grid}_${method}.${datestring}.nc
map_${atm_grid}_to_${lnd_grid}_${method}.${datestring}.nc
map_${lnd_grid}_to_${atm_grid}_${method}.${datestring}.nc
map_${lnd_grid}_to_${ocn_grid}_${method}.${datestring}.nc
map_${ocn_grid}_to_${lnd_grid}_${method}.${datestring}.nc |
where ${ocn_grid}, ${atm_grid}, and ${lnd_grid} are the names of the ocean and atmos grid provided above in the --nm_src and --nm_dst arguments, and ${datestring} is the date string provided above in the --dt_sng argument to ncremap. The ${method} can be monotr, highorder, mono, intbilin, aave, blin, ndtos, nstod, and patc. (TODO: What do all these mean?) If using the tri-grid option, the land grid files will be created from the ${lnd_grd}.
4. Generate domain files
Domain files are needed by the coupler and the land model at runtime. The land model uses the mask to determine where to run and the coupler use the land fraction to merge fluxes from multiple surface types to the atmosphere above them. Domain files are created from the mapping files created in the previous step, using a tool provided with CIME in ${e3sm_root}/cime/tools/mapping/gen_domain_files. This directory contains the source code for the tool (in Fortran 90) and a Makefile. Cloning E3SM is now required to obtain code within the distribution. To clone E3SM,
| Code Block |
|---|
# Checkout E3SM code
git clone git@github.com:E3SM-Project/E3SM.git |
Assuming ${e3sm_root} is set and the location of TempestRemap binaries is added to ${PATH}, the following script should generate the appropriate domain files:
| Code Block |
|---|
#!/bin/bash
# Setup environment
source /global/project/projectdirs/acme/software/anaconda_envs/load_latest_e3sm_unified.sh
e3sm_root=${HOME}/codes/e3sm/branches/master
# Build gen_domain tool
export USER_FFLAGS='-traceback -g -O0'
gen_domain=${e3sm_root}/cime/tools/mapping/gen_domain_files/gen_domain
cd `dirname ${gen_domain}`/src
../../../configure --macros-format Makefile --mpilib mpi-serial
(. ./.env_mach_specific.sh ; gmake)
# Set paths to mapping files
mapping_root="/global/homes/b/bhillma/cscratch/e3sm/grids/ne4"
ocn_grid_name=oQU240
atm_grid_name=ne4np4
lnd_grid_name=${atm_grid_name}
# run domain generation tool (from output directory)
domain_root=${mapping_root}
mkdir -p ${domain_root} && cd ${domain_root}
for target_grid_name in ${lnd_grid_name} ${atm_grid_name}; do
# Find conservative mapping files, use the latest file generated
map_ocn_to_target=`ls ${mapping_root}/map_${ocn_grid_name}_to_${target_grid_name}_monotr.*.nc | tail -n1`
# Run domain tool code
${gen_domain} -m ${map_ocn_to_target} -o ${ocn_grid_name} -l ${target_grid_name}
done
|
If this command is successful, it should produce many domain files in ${output_root}/domain_file that look something like
| Code Block |
|---|
domain.lnd.${atm_grid}_${ocn_grid}.${datestring}.nc
domain.ocn.${atm_grid}_${ocn_grid}.${datestring}.nc
domain.ocn.${ocn_grid}.${datestring}.nc
domain.lnd.${lnd_grid}_${ocn_grid}.${datestring}.nc
domain.ocn.${lnd_grid}_${ocn_grid}.${datestring}.nc |
5. Generate topography file
Requirements:
- High resolution USGS topography file named USGS-topo-cube3000.nc (located in the CESM inputdata server here; note this is on a 3 km cubed sphere grid)
- SCRIP-format atmosphere grid file
- topography tool (
components/cam/tools/topo_tool/cube_to_target) - homme_tool (same tool used to create SCRIP files described above).
Topography needs to be interpolated from a high resolution USGS file, and then doctored up a bit to allow the model to run stably with the new topography. The tool chain used to compute topography is documented in the following paper:
There are three main steps in this process:
Step (a) downscale USGS topography data to the atmosphere grid
Step (b) apply dycore-specific smoothing to reduce the amount of gravity wave noise if the topography is too rough. This is a tunable parameter, and a rigorous analysis probably needs to be performed to determine the optimal tuning parameter for future very-high-resolution atmosphere grids. For pg2 grids there is the extra step of generating consistent topo data on both the GLL (dynamics) and pg2 (physics) grid.
Step (c) based on the smoothed topography, compute the SGH and SGH30 surface roughness fields used by some parameterizations
Steps (a) and (c) use the topography tool located at components/cam/tools/topo_tool/cube_to_target in the E3SM repo. This requires the SCRIP dualgrid file.
Step (b) requires running the homme_tool's smoothing algorithm. The namelist should be edited to specify the resolution (NE value) or grid file (RRM) grid and the amount of smoothing
(TODO: allow running this script with command line arguments instead of manual editing so that the task can be automated).
For physics and dynamics on the GLL grid (E3SM V1): see (move text below to separate page)
For "pg2" grids (E3SM v2): see the instructions on Special Considerations for FV Physics Grids
Step (a)
To begin Step (a) with a SCRIP file in hand, interpolate the high resolution USGS data. The source topography file is the USGS high resolution topography on a 3 km cube sphere grid. This data exists on the CESM inputdata server here. The data has also been downloaded to NERSC computing facilities and is located here:
| Code Block |
|---|
/project/projectdirs/acme/inputdata/atm/cam/hrtopo/USGS-topo-cube3000.nc |
Interpolation to the target grid is done using the cube_to_target tool. This tool has recently been modified to accept arguments via the command line, which prevents having to edit the source code and recompile to run the different steps of a) interpolating the USGS topography to the target grid and b) calculating the subgrid surface roughness based on the interpolated topography and one that has been smoothed by the dynamical core. If successful, this script prints a clean exit with "Done running cube_to_target."
| Code Block |
|---|
#!/bin/bash
grid_name=ne4
output_root=/gscratch/bhillma/e3sm/grids/${grid_name}
atm_scrip_file="${output_root}/ne4np4b_scrip.nc"
# Set paths
e3sm_root="${HOME}/codes/e3sm/branches/master"
datestring=`date +'%y%m%d'`
cube_to_target_root=${e3sm_root}/components/cam/tools/topo_tool/cube_to_target
input_topography_file=/project/projectdirs/acme/inputdata/atm/cam/hrtopo/USGS-topo-cube3000.nc
output_topography_file=${output_root}/topo/USGS_${grid_name}_unsmoothed_${datestring}.nc
# Get machine-specific modules
${e3sm_root}/cime/tools/configure && source .env_mach_specific.sh
# build the code
cd ${cube_to_target_root}
export FC=ifort INC_NETCDF=${NETCDF_DIR}/include LIB_NETCDF=${NETCDF_DIR}/lib
make clean && make
echo "Done building cube_to_target."
# run the code
echo "Run cube_to_target to interpolate topography..."
mkdir -p `dirname ${output_topography_file}`
${cube_to_target_root}/cube_to_target \
--target-grid ${atm_scrip_file} \
--input-topography ${input_topography_file} \
--output-topography ${output_topography_file}
echo "Done running cube_to_target." |
Step (b)
The unsmoothed topography then needs to be smoothed using the smoothing operator in HOMME. This is done using the homme_tool utility. Follow the instructions above in the SCRIP section to configure and build this tool. A sample namelist for the topo smoothing option is given in components/homme/test/tool/toposmooth_gll.nl
- For cubed sphere grids:
- Set resolution via
set ne = 4(for ne4 example here). - Set
smooth_phis_numcycle=16. (adjustable paramemter, lower for less smoothing) - Set smooth_phis_nudt. This is a resolution dependent parameter, recommend value for cubed sphere grids is 28e7 * ( 30/NE)**2
- Set resolution via
- For RRM grids, we use the tensor laplace operator so that the amount of smoothing is proportional to the resolution
- ne=0, mesh_file=/path/to/exodus-rrm-grid-file.g
- Set
smooth_phis_numcycle=12. (adjustable paramemter, lower for less smoothing) - smooth_phis_nudt = 4e-16.
- Set infilenames=/path/to/unsmoothed/PHISdata
After running the tool, the output will be named "geos", in the file phis-smoothed1.nc, along with other unnecessary fields. To convert this file into an EAM topo file:
- ncks -O -v geos,lat,lon phis-smoothed1.nc phis-smoothed2.nc
- ncrename -v geos,PHIS phis-smoothed2.nc
Step (c)
The topography tool then needs to be re-ran to compute the subgrid surface roughness. This is done by passing an extra argument to the topography tool with the smoothed topography output from the standalone HOMME run above:
| Code Block |
|---|
# Get machine-specific modules
e3sm_root=${HOME}/codes/e3sm/branches/update-topo-tool
${e3sm_root}/cime/tools/configure && source .env_mach_specific.sh
# Set paths; datestring will probably be different if the workflow takes multiple days!
datestring=`date +'%y%m%d'`
atm_scrip_file="${output_root}/templates/ne4np4b_scrip_${datestring}.nc"
input_topography_file="${inputdata_root}/atm/cam/hrtopo/USGS-topo-cube3000.nc"
output_topography_file="${output_root}/topo/USGS_ne4np4_consistentSGH_${datestring}.nc"
smoothed_topography_file="${output_root}/topo/USGS_ne4np4_smoothed_${datestring}.nc"
# Run the code
${cube_to_target_root}/cube_to_target \
--target-grid ${atm_scrip_file} \
--input-topography ${input_topography_file} \
--output-topography ${output_topography_file} \
--smoothed-topography ${smoothed_topography_file} |
...
| Warning | ||
|---|---|---|
| ||
NERSC Directory Change Notice Due to project's name change at NERSC from 'ACME' to 'E3SM' and NERSC’s file system update, the directory ' |
Table of Contents
| Table of Contents |
|---|
Child Pages
Child pages (Children Display)
The purpose of this page is to document the procedure for adding support for new atmosphere grids. The process should be the same for new uniform resolutions as well as for new regionally-refined meshes, although some settings will need to be changed for new regionally-refined mesh configurations. This page is a work in progress, and will be updated as this process is refined and (eventually) made more automated. This documentation is an update of a document written by Mark Taylor and Colin Zarzycki, available as a Google Doc here. Linking to a related page: /wiki/spaces/NGDNA/pages/915408074.
For the purpose of this step-by-step guide, we will walk through each step using an example of regenerating the ne4 grid that is already currently supported. For a new grid (i.e., ne512), just replace ne4 with the desired grid identifier. This is determined in Step 1 Generate a new "grid" file below, and the rest should follow.
For this example, I will assume a few environment variables, which we should set now. First, determine where to put the generated output files. For big grids, this can take up quite a bit of space, so choose a location with sufficient space. I have a symlink to my CSCRATCH directory on NESRC in my home directory, so I will put grid files there for this example:
| Code Block |
|---|
output_root=${HOME}/cscratch/e3sm/grids/ne4
mkdir -p ${output_root} |
Types of Atmosphere grid metadata files
See CAM-SE Grid Overview for description of the spectral elements, GLL nodes, subcell grid and dual grid.
- Exodus file: "ne4.g". This is a netcdf file following Exodus conventions. It gives the corners of all elements on the sphere and their connectivity. It is independent of the polynomial order used inside the element ("np").
- This file is used by TempestRemap (TR) to generate mapping files. The polynomial order is a command line option and the GLL nodes are internally generated by TR.
- latlon file: "ne4np4_latlon.nc". This file contains a list of all the GLL nodes in the mesh (in latitude/longitude coordinates). The list of GLL nodes must be in the the internal HOMME global id ordering, matching the ordering used in CAM and EAM native grid output. It also contains the connectivity of the GLL subcell grid.
- This file is used by CAM's interpic_new utility, and graphics programs Paraview and Visit when plotting native grid output.
- SCRIP file: "ne4np4_scrip.nc". This file contains a description of the GLL dual grid, in the format used by the original incremental remap tool SCRIP. It includes the locations of the GLL nodes and artificial bounding polygons around those nodes. Ideally the spherical area of each polygon will match the GLL weight ("exact GLL areas"), but not all tools can achieve exact areas. Inexact areas does not impact the accuracy of the resulting mapping algorithm, it just means that mass will not be exactly conserved by the mapping algorithm.
- Needed in E3SMv2 configurations by the cube_to_target topography dataset generation tool.
- Historically used by CESM/E3SMv1 to create ocean/atm mapping files (exact GLL areas are important here, to conserve fluxes)
- Used by the regional and sub-gridscale regridding post-processing tools ( https://acme-climate.atlassian.net/wiki/spaces/SIM/pages/754286611/Regridding+E3SM+Data+with+ncremap, http://nco.sf.net/nco.html#sgs)
...
Step-by-step guide
1. Generate a new "grid" file
Requirements: TempestRemap
In order to generate mapping files to do interpolation between a new atmosphere grid and the other component grids, we need to create a file that describes the atmosphere grid representation. The TempestRemap tool (https://github.com/ClimateGlobalChange/tempestremap) can be used to recreate the internal representation of the spectral element grids used in HOMME. The grid mesh file will be saved in an "exodus" type file (with a .g extension).
TempestRemap needs to be built from source from the Github repository. This is straightforward on Cori and Edison. Note that a parallel version of netCDF should be used. This can be accomplished on cori/edison be executing module swap cray-netcdf cray-netcdf-hdf5parallel or by sourcing an env_mach_specific.sh from a working case on the target machine before building. Then,
| Code Block |
|---|
git clone https://github.com/ClimateGlobalChange/tempestremap.git
cd tempestremap
make -f Makefile.gmake
cd .. |
This will build TempestRemap into tempestremap/bin. For the rest of the tutorial, assume that the environment variable ${tempest_root} points to this directory:
| Code Block |
|---|
tempest_root=${PWD}/tempestremap |
We can generate an ne4 mesh easily now:
| Code Block |
|---|
${tempest_root}/bin/GenerateCSMesh --alt --res 4 --file ${output_root}/ne4.g |
For RRM grids, SQuadGen should be used (available in a separate repository on Github here). This will be documented in another page (TODO).
The Exodus file contains only information about the position of the spectral element on the sphere. For SE aware utilities such as TempestRemap, they can use the polynomial order and the reference element map to fill in necessary data such as the locations of the nodal GLL points. For non-SE aware utilities, we need additional meta data, described in the next section.
2A. Generate "dual grid" mesh files for E3SM v2 "pg2" grids
Requirements:
- exodus mesh file
- TempestRemap
In E3SM v2, we will be switching to running physics on a FV pg2 grid and mapping files between the atmosphere and other components will be FV to FV type maps using the pg2 grids. These can be generated by TempestRemap directly from the exodus mesh file:
| Code Block |
|---|
${tempest_root}/bin/GenerateVolumetricMesh --in ne4.g --out ne4pg2.g --np 2 --uniform
${tempest_root}/bin/ConvertExodusToSCRIP --in ne4pg2.g --out ne4pg2.scrip.nc |
2B. Generate "dual grid" mesh files (SCRIP and lat/lon format) for E3SM v1 "np4" GLL grids
Requirements:
- exodus mesh file
- Matlab or Fortran utility.
Note: in E3SM v2, we will be switching to running physics on a FV pg2 grid and mapping files between the atmosphere and other components will be FV to FV type maps using the pg2 grids. The spectral element "np4" grid is still used internally by the dynamics and for initial conditions, so the metadata described in this section is still needed for some analysis and preparation of initial conditions.
Mapping files used by the coupler for mapping fluxes between SE "np4" and FV grids should be generated with TempestRemap and only need the exodus grid description (which provides the locations of the corners of the quadrilaterals that form the elements of the cube-sphere mesh) generated in Step 1 above. However, a handful of pre- and post-processing tools require a finite-volume equivalent of the spectral element grid (these tools include the surface roughness calculation in the land tool cube_to_target, ESMF mapping files used for interpolating land surface data to target grid, and post-processing regional and subgrid remapping tools). We refer to this finite-volume description of the SE grid as the "dual grid" to the SE grid (see page describing atmosphere grids in more detail here).
The dual grid is generated using separate tools that draw polygons around the GLL nodes, and optionally optimize the area contained within each polygon to match the GLL weights. The resulting grid is saved in netCDF files that contain the grid point locations, area weights, and areas associated with each nodal value. In the SE dycore, this data depends on the reference element map (how the finite elements are mapped into the unit square reference element). The SE dycore supports two such reference element maps: the older "gnomonic equal angle" map, and an "element-local" map. RRM grids are required to use the "element-local" map. Cubed-sphere grids can use either map (chosen via namelist option), with the existing compsets in E3SM v1 all using the gnomonic equal area map. For E3SM v2, all new grids (cubed-sphere and RRM) should use the element-local map.
There are three separate codes that can create a dual grid from our SE grids:
- Matlab code. This code produces the best SCRIP files, with nice (non-starlike) control volumes around each GLL node, with exact areas (spherical area = GLL weight). But the serial code is painfully slow and thus is only practical for grids with up to 100,000 elements. This code only supports the element-local map. Noel Keen profiled the matlab script and found that at least for ne30 case, ~90% of the time is spent in computing the area of a polygon which is called millions of times. There may be some simple things to try to improve outside of rewriting.
- Fortran code. This utility is included with standalone HOMME. It runs quickly, in parallel, and produces exact GLL areas for cubed-sphere meshes. It supports both element maps via namelist options. But for RRM meshes it produces suboptimal control volumes (including star shaped control volumes) with only approximate areas (spherical area ≠ GLL weight). The resulting RRM SCRIP file is thus not acceptable for generating mapping files for coupled simulations, but they should be fine for generating surface roughness fields.
- NCAR utility. Will update when we find out more about this.
For the majority of use cases, optimizing the area and shape of each polygon in the dual grid is probably not necessarily, and a simple (and fast) approach can be taken using the Fortran code (plus some NCL scripts), which is packaged with HOMME. For conservative remapping, optimal areas that exactly match the GLL weights probably is required, and the slower Matlab code will most likely need to be used (NOTE: we plan to rewrite the Matlab code in a compiled language to work with higher resolution grids, but for the time being we will work with the less accurate Fortran/NCL codes for the few use cases we still need the SCRIP files for).
The _latlon file is a separate grid descriptor file that contains just a simple list of coordinates of all GLL nodes and is easy to produce. Either the Fortran+NCL or the Matlab utilities can produce this file.
The Fortran Code
To run the fortran code on a E3SM supported platform: (updated 2020/5/25 with updated tool from https://github.com/E3SM-Project/E3SM/pull/3593 )
- Be sure your environment matches the software environment loaded by E3SM by executing the output of this command: e3sm/cime/scripts/Tools/get_case_env
- Use cmake to configure and compile standalone HOMME. On a supported platform with the CIME environement, this should work out-of-the-box. See e3sm/cime/components/homme/README.cmake
- compile the HOMME tool utility:
- cd /path/to/workingdir
- make -j4 homme_tool
- executable: /path/to/workingdir/src/tool/homme_tool
- Edit e3sm/components/homme/test/tool/namelist/template.nl and specify the grid resolution or RRM file
- For ne512, this would be
set ne = 512. For RRM grids, leavene = 0, but will need to edit where the exodus grid file comes from - for non-RRM grids using the older E3SM v1 dycore, add cubed_sphere_map=0 to template.nl
- For ne512, this would be
- See e3sm/components/homme/test/tool/test.job for examples of how to run the script and then use an NCL utilities to process the tool output into SCRIP and latlon formats.
Specific details for running at NERSC on Cori(knl):
- Create a batch script hange "account" in the sbatch directives at the top of the script. For example, set
#SBATCH --account=e3sm - cmake -C /path/to/e3sm/components/homme/cmake/machineFiles/cori-knl.cmake -DPREQX_NP=4 /path/to/workingdir
- Make sure a working NCL is in your PATH. On Cori, add the following to the script:
module load ncl.
With these changes, run makegrid.job TWICE to build and then run the code. The first pass builds HOMME, and the second pass runs HOMME and the NCL utilities. In a HPC environment, the second pass needs to submitted to the batch queue via sbatch -C knl makegrid.job. This will create the grid descriptor files in ${HOME}/scratch1/preqx/template.
The Matlab Code
The tool chain described below only supports the element-local map. As noted above, this method produces the highest-quality output but is extremely slow (days to weeks) to run. Our preferred mapping algorithms (Transition to TempestRemap for Atmosphere grids) also only support the element-local map.
Creating the grid descriptor files for RRM grids and cubed-sphere grids using the element local map is accomplished using a matlab code written for this process, available in the PreAndPostProcessingScripts repo. First, checkout the code:
| Code Block |
|---|
# Checkout code
git clone git@github.com:ACME-Climate/PreAndPostProcessingScripts.git
# Set variable so we can refer to this directly below
PreAndPostProcessingScripts=${PWD}/PreAndPostProcessingScripts |
Now we can run the code by executing the run_dualgrid.sh script from the ${PreAndPostProcessingScripts}/regridding/spectral_elements_grid_utilities/compute_dualgrid directory. Note that a working Matlab environment is required to run the code. On NERSC, this can be accomplished by simply executing module load matlab before running the run_dualgrid.sh script.
| Code Block |
|---|
grid_name=ne4
mesh_file=${output_root}/ne4.g
module load matlab
cd ${PreAndPostProcessingScripts}/regridding/spectral_elements_grid_utilities/compute_dualgrid
./run_dualgrid.sh ${grid_name} ${mesh_file} |
This will create two files in the compute_dualgrid directory: a "latlon" and a "scrip" grid descriptor file. Move these to wherever you want to keep your grid files:
| Code Block |
|---|
mv *_latlon.nc ${output_root}
mv *_scrip.nc ${output_root} |
3. Generate mapping files
Requirements:
- TempestRemap
- ESMF_RegridWeightGen
- ncremap
- grid descriptor files for each component that exists on a different grid
(atmosphere, ocean, possibly land if on a different grid than the atmosphere)
In order to pass data between different components at runtime, a set of mapping files between each component is generated offline. These mapping files will also be used in Step 4 below (generating domain files).
For "pg2" grids used in E3SM v2, the recommended mapping files are slightly different and still a work in progress. See Transition to TempestRemap for Atmosphere grids
TempestRemap and ESMF are the backends that generate the mapping weights, but this is all nicely encapsulated using ncremap. Tempest is the preferred method for creating mapping files. The ncremap function, calling TempestRemap or ESMF, will decide which of these two tools to use based on the atmospheric input file. If the *.scrip file was used, then ESMF will be called. If the *.g file was used, then TempestRemap will be called. The ESMF tools are adequate for making atmosphere-only-type component sets for E3SM, but this tool is less conservative than TempestRemap. If you are making grids for a coupled run, then TempestRemap should be used.
The easiest way to make sure ncremap is up to date and will work for this workflow, we can source the ESMF-Unified conda environment. Instructions for supported machines can be found here. For Edison and Cori, it is as simple as:
| Code Block |
|---|
source /global/project/projectdirs/acme/software/anaconda_envs/load_latest_e3sm_unified.sh |
For unsupported machines you may need to install the conda environment via:
| Code Block |
|---|
conda install -c conda-forge -c e3sm -c cdat e3sm-unified |
TempestRemap was built in Step #1 above. We need to tell ncremap where to find our TempestRemap binaries, so we need to append to our PATH:
| Code Block |
|---|
PATH=${tempest_path}/bin:${PATH} |
And now we can use ncremap to generate ALL the needed mapping files between two grids, in this example the ne4 atmosphere and the oQU240 ocean grid (for the moment, we will put the land on the same grid as the atmosphere):
| Code Block |
|---|
atm_grid_file=ne4.g
ocn_grid_file=/project/projectdirs/acme/inputdata/cpl/gridmaps/oQU240/ocean.QU.240km.scrip.151209.nc
cd ${output_root} && ncremap -P mwf -s $ocn_grid_file -g $atm_grid_file --nm_src=oQU240 --nm_dst=ne4np4 --dt_sng=20181114 |
In the above code, the "-P mwf" option triggers a specific procedure that invokes multiple ncremap commands which each invoke multiple Tempest commands.It produces 3 mapping files each for a2o and o2a. This multitude of mapping files for each component→component case are needed because fluxes need conservative mapping and states need non-conservative mapping. Because "-P mwf" results in a great deal of output being flushed to screen, it is mostly suppressed by default. To see all the output or to figure out how to run all the nco and tempest remap commands one-by-one, add "--dbg_lvl=2" to the ncremap command. This will print the commands but not execute them. The user can then use one of the printed commands, again with "--dbg_lvl=2" to see the actual commands being sent to Tempest.
If this command is successful, it should produce many mapping files in ${output_root}, that look something like
| Code Block |
|---|
map_${atm_grid}_to_${ocn_grid}_${method}.${datestring}.nc
map_${ocn_grid}_to_${atm_grid}_${method}.${datestring}.nc
map_${atm_grid}_to_${lnd_grid}_${method}.${datestring}.nc
map_${lnd_grid}_to_${atm_grid}_${method}.${datestring}.nc
map_${lnd_grid}_to_${ocn_grid}_${method}.${datestring}.nc
map_${ocn_grid}_to_${lnd_grid}_${method}.${datestring}.nc |
where ${ocn_grid}, ${atm_grid}, and ${lnd_grid} are the names of the ocean and atmos grid provided above in the --nm_src and --nm_dst arguments, and ${datestring} is the date string provided above in the --dt_sng argument to ncremap. The ${method} can be monotr, highorder, mono, intbilin, aave, blin, ndtos, nstod, and patc. (TODO: What do all these mean?) If using the tri-grid option, the land grid files will be created from the ${lnd_grd}.
4. Generate domain files
Domain files are needed by the coupler and the land model at runtime. The land model uses the mask to determine where to run and the coupler use the land fraction to merge fluxes from multiple surface types to the atmosphere above them. Domain files are created from the mapping files created in the previous step, using a tool provided with CIME in ${e3sm_root}/cime/tools/mapping/gen_domain_files. This directory contains the source code for the tool (in Fortran 90) and a Makefile. Cloning E3SM is now required to obtain code within the distribution. To clone E3SM,
| Code Block |
|---|
# Checkout E3SM code
git clone git@github.com:E3SM-Project/E3SM.git |
Assuming ${e3sm_root} is set and the location of TempestRemap binaries is added to ${PATH}, the following script should generate the appropriate domain files:
| Code Block |
|---|
#!/bin/bash
# Setup environment
source /global/project/projectdirs/acme/software/anaconda_envs/load_latest_e3sm_unified.sh
e3sm_root=${HOME}/codes/e3sm/branches/master
# Build gen_domain tool
export USER_FFLAGS='-traceback -g -O0'
gen_domain=${e3sm_root}/cime/tools/mapping/gen_domain_files/gen_domain
cd `dirname ${gen_domain}`/src
../../../configure --macros-format Makefile --mpilib mpi-serial
(. ./.env_mach_specific.sh ; gmake)
# Set paths to mapping files
mapping_root="/global/homes/b/bhillma/cscratch/e3sm/grids/ne4"
ocn_grid_name=oQU240
atm_grid_name=ne4np4
lnd_grid_name=${atm_grid_name}
# run domain generation tool (from output directory)
domain_root=${mapping_root}
mkdir -p ${domain_root} && cd ${domain_root}
for target_grid_name in ${lnd_grid_name} ${atm_grid_name}; do
# Find conservative mapping files, use the latest file generated
map_ocn_to_target=`ls ${mapping_root}/map_${ocn_grid_name}_to_${target_grid_name}_monotr.*.nc | tail -n1`
# Run domain tool code
${gen_domain} -m ${map_ocn_to_target} -o ${ocn_grid_name} -l ${target_grid_name}
done
|
If this command is successful, it should produce many domain files in ${output_root}/domain_file that look something like
| Code Block |
|---|
domain.lnd.${atm_grid}_${ocn_grid}.${datestring}.nc
domain.ocn.${atm_grid}_${ocn_grid}.${datestring}.nc
domain.ocn.${ocn_grid}.${datestring}.nc
domain.lnd.${lnd_grid}_${ocn_grid}.${datestring}.nc
domain.ocn.${lnd_grid}_${ocn_grid}.${datestring}.nc |
5. Generate topography file
Requirements:
- High resolution USGS topography file named USGS-topo-cube3000.nc (located in the CESM inputdata server here; note this is on a 3 km cubed sphere grid)
- SCRIP-format atmosphere grid file
- topography tool (
components/cam/tools/topo_tool/cube_to_target) - homme_tool (same tool used to create SCRIP files described above).
Topography needs to be interpolated from a high resolution USGS file, and then doctored up a bit to allow the model to run stably with the new topography. The tool chain used to compute topography is documented in the following paper:
There are three main steps in this process:
Step (a) downscale USGS topography data to the atmosphere grid
Step (b) apply dycore-specific smoothing to reduce the amount of gravity wave noise if the topography is too rough. This is a tunable parameter, and a rigorous analysis probably needs to be performed to determine the optimal tuning parameter for future very-high-resolution atmosphere grids. For pg2 grids there is the extra step of generating consistent topo data on both the GLL (dynamics) and pg2 (physics) grid.
Step (c) based on the smoothed topography, compute the SGH and SGH30 surface roughness fields used by some parameterizations
Steps (a) and (c) use the topography tool located at components/cam/tools/topo_tool/cube_to_target in the E3SM repo. This requires the SCRIP dualgrid file.
Step (b) requires running the homme_tool's smoothing algorithm. The namelist should be edited to specify the resolution (NE value) or grid file (RRM) grid and the amount of smoothing
(TODO: allow running this script with command line arguments instead of manual editing so that the task can be automated).
For physics and dynamics on the GLL grid (E3SM V1): see Topography generation for GLL grids
For "pg2" grids (E3SM v2): see the instructions on Special Considerations for FV Physics Grids
6. Generate and spin-up a new atmosphere initial condition
...
- land grid descriptor file in SCRIP format
- ESMF_RegridWeightGen
- geographic distribution for each land surface type along with grid descriptor files for each of those surface types
- mkmapdata.sh (found in components/clm/tools/shared/mkmapdata/)
- mksurfdata.pl (found in components/clm/tools/clm4_5/mksurfdata_map/)
...