800m cubed topo generation from GMTED2010 15s DEM

Owned by Jishi Zhang
Last updated: Mar 15, 2024

Abstract: There is a small task out of LLNL LDRD-SI ``Climate Resilience for National Security’’ — we are curious if we can run SCREAM RRM at 100m. However, the existing USGS cubed topo is all in 3 km processed from the USGS GTOPO30 30 arch-second (1 km) source DEM (10.5194/gmd-8-3975-2015). Therefore, we need to generate new cubed topo using ultra-high resolution source DEM data. Some of us are interested in the process of generating new global uniform cubed topo used for whatever new grids with dx < 3 km @Mark Taylor, @Peter Bogenschutz, @Jishi Zhang. Specifically, this page documents the practice on LC of generating 800m cubed topo using 500m / 250m USGS GMTED2010 source DEM dataset.

Note that this practice is still far away from our original goal (100m RRM). Reasons include 1) I haven’t found any global DEM dataset with resolution higher than 250m, and thus stitching of different datasets is required for dx < 250m; 2) OOM issue from 250m source DEM to cubed topo (``bin_to_cube’’ step) on both LC and perlmutter; 3) the 100m RRM simulation has its own issues (irrelevant to new topos) that have not been addressed.   

 

Step 1. Prepare lat-lon topo source DEM. Variables include elevation (``htopo’’) and land fraction (``landfract’’)

  • 500m GMTED2010 15 arch-second source DEM：

  1. Got the processed data from Jos van Geffen <geffen@knmi.nl> and this data seems to be processed by Maarten Sneep <maarten.sneep@knmi.nl> from the file history. This file is already on global lat-lon netcdf format. It includes the land fraction from MODIS MOD44W land-water mask.

  2. Re-class land fraction according to https://github.com/NCAR/Topo/tree/old/create_netCDF_from_rawdata/create_gmted2010_modis.py , and re-format this file:

; ---24-3: 0,5,6,7,15->0; others=1 ncap2 -O -s 'where(LandWater == 2 || LandWater == 3 || LandWater == 4) LandWater = 1;' S5P_OPER_REF_DEM_15_00000000T000000_99999999T999999_20160111T104226.nc S5P_OPER_REF_DEM_15_00000000T000000_99999999T999999_20160111T104226.landfrac24-3.nc ncap2 -O -s 'where(LandWater != 1) LandWater = 0;' S5P_OPER_REF_DEM_15_00000000T000000_99999999T999999_20160111T104226.landfrac24-3.nc S5P_OPER_REF_DEM_15_00000000T000000_99999999T999999_20160111T104226.landfrac24-3.nc ncks -5 S5P_OPER_REF_DEM_15_00000000T000000_99999999T999999_20160111T104226.NCL_htopo.nc S5P_OPER_REF_DEM_15_00000000T000000_99999999T999999_20160111T104226.NCL_24-3.nc ; ---after NCL ncks -O -5 S5P_OPER_REF_DEM_15_00000000T000000_99999999T999999_20160111T104226.NCL_landfrac24-3.nc S5P_OPER_REF_DEM_15_00000000T000000_99999999T999999_20160111T104226.NCL_landfrac24-3.nc ncks -A -v landfract S5P_OPER_REF_DEM_15_00000000T000000_99999999T999999_20160111T104226.NCL_landfrac24-3.nc S5P_OPER_REF_DEM_15_00000000T000000_99999999T999999_20160111T104226.NCL_24-3.nc

 

  • 250m GMTED2010 7.5 arch-second source DEM：

  1. Got the source .adf tiles from https://topotools.cr.usgs.gov/gmted_viewer/gmted2010_global_grids.php

  1. Combine those tiles to a global lat-lon netcdf using gis tool:

gdal_translate -of netCDF /p/lustre2/zhang73/DATA/DEM/be75_grd /p/lustre2/zhang73/DATA/DEM//be75_grd_1.nc

  1. Other processing: a) no polar regions, stitched from 500m GMTED2010 DEM (first remapped to 250m resolution); b) fill Greenland from 500m GMTED2010 DEMl; c) use landfract interpolated from 500m GMTED2010 DEM.

cdo --no_remap_weights remapbil,r172800x86400 S5P_OPER_REF_DEM_15_00000000T000000_99999999T999999_20160111T104226.NCL_24-3.nc S5P_OPER_REF_DEM_15_00000000T000000_99999999T999999_20160111T104226.NCL_24-3.r172800x86400.nc

grid_WL.topo.DEM.src_glb.stitch.ncl

 

 

Step 2. bin_to_cube:

  • build issues

  1. cannot find netcdf.inc — assgin FC to ifort in Makefile, and add a line ``FFLAGS := -I$(INC_NETCDF)'' after defining ``INC_NETCDF'' in Makefile

  2. warning: libnetcdf.so.19, needed by /usr/tce/backend/installations/linux-rhel8-x86_64/intel-2021.6.0/netcdf-fortran-4.6.0-gbg3y7c2xjnda55wkwwzupgzgcmevm3w/lib/libnetcdff.so, may conflict with libnetcdf.so.15 — need to assign the lib path including so.19  ``ifort -o bin_to_cube bin_to_cube.o shr_kind_mod.o -L/path/to/correct/netcdf/lib -lnetcdff ...’’

  • source code changes:

Needed to copy some of codes in Peter Lauritzen’s repo to the topo_tool in E3SM repo, e.g., the E3SM topo_cool’s bin_to_cube version removed several lines that dividing terr_cube by weight (seems that it’s deleted in the commit “Remove LANDM_COSLAT from topo tools” https://github.com/E3SM-Project/E3SM/commit/3ab98d3cef26a480850a3c1f45370b3428c25995 ). Except those modifications, I just needed to change the format of the written netcdf file from “NF_64BIT_OFFSET” to  “NF_64BIT_DATA”.

  • 250m source DEM to cubed topo failed due to OOM. It requires ~638G to cube48000 (200m), ~507G to cube40000 (250m), ~316G to cube24000 (400m), not enough for both LC and perlmutter.

 

 

Step 3. cube_to_target:

  • build issues:

Makefile:

build command:

  • source code changes:

Needed to increase ``jall'' and ``jmax_segments'' (set jall = default *100000, jmax_segments = default *2). I didn’t realize the array out-of-bounds for the default jmax_segments when the process freezed, until the indexes of the array were printed.

 

 

Preliminary analysis for the new 800m cubed topo: 

We have briefly compared the previous USGS_topo_cube3000.nc with the new topo cube using the 15s DEM (S5P_OPER_REF_DEM_15_24-3_cube12000.nc), which seems to be reasonable. The cube12000 has a much smaller SGH30 due to the smaller deviations from 500m (15s source DEM) to 800m (cube12000), as compared to the deviations from 1km (30s DEM) to 3km (cube3000). Although SGH30 hasn’t affect our tests as we are using SCREAMv0 codebase which doesn’t use TMS.

unknown.png

 

Besides, we have looked for a RRM test using the new cube12000 topo compared to the default one. This is based on a hindcast Peter B is working on, using a 800m RRM over California. We found that using the new topo brought some more detail to certain areas, such as many east-west gulches in the Sierra Nevada—although the difference in general didn't seem to be very noticeable.

image-20240313-063519.png

 

Location of source DEMs and the 800m cubed topo:

The 15s + 7.5s DEMs and the 800m topo cube files are put on Perlmutter:

/pscratch/sd/z/zhang73/grids2/topo15s/

/pscratch/sd/z/zhang73/grids2/topo7.5s/

  • 15 arch-second (500m) source DEM GMTED2010:

  • 800m cubed topo based on 15s DEM GMTED2010:

  • 7.5 arch-second (250m) source DEM GMTED2010: