Document issues with our mapping file generation tool chain at high resolution. Instructions on using MBtempest: Offline remapping workflow with mbtempest
Status
For PG2 grids, we will mapping FV-FV. For monotone conservative linear maps, the only choice is cell integrated using a piecewise constant reconstruction. Options for producing these maps include ESMF, TempestRemap (TR), Moab with TR agorithms (MBTR) and NCO. All codes produce similar max and l2 errors when evaluated with the analytic vortex test field.
As of
NCO: good results, fastest code. ~1.5H using 4 cores (4 threads) Cori-Haswell node. oRRS18to6 → ne1024pg2
TR: good results with code from and with the “--correct_areas” option. Very slow: 15H (1 core) oRRS18to6 → ne1024pg2
MBTR: maps are bad
ESMF: maps for ne1024 are bad.
For NP4 grids,:
TR from : most maps are good, some combinations still produce crashes.
MBTR: maps are bad. is producing bad weights (NaNs or >> 1 ).
...
Analysis Mapping files
Mapping files are needed for remapping native grid output to lat/lon grids for conventional analysis tools. Often exact conservation is not required making TR’s integrated bilinear a good option: it’s accurate, has built in downscaling and is monotone so it wont introduce spurious oscillations making it good for plotting.
TR “intbilin”
several issues with loss of precision documented on HOMME-NH dycore scaling to 1km /wiki/spaces/NGDNA/pages/831554328
SE GLL to SE GLL
These mapping files are mostly used to generate initial conditions. Remapping the cam.i file from a spunup NE30 simulation will generate a reasonably well spunup initial condition file for higher resolutions.
TR “mono” (with code fixes from )
ne30np4 → ne256np4
NaNs in weights.
issue: ncks --chk_map and AnalyzeMap ignore NaNs when computing weight min/max.
issue: AnalyzeMap ignores NaNs when computing row sum
Status: NOT USABLEweights are good. row sum error 1.3e-15, conservation error: 5e-14.
Status: GOOD
ne30np4->ne512np4
NaNs in weights.
Status: NOT USABLEGOOD
MBTR “mono”
ne30np4->ne256np4
--monotonic 1: mapping weights have NaNs
--monotonic 3 --noconserve (“intbilin” option) weights are bad, max = 2.5
Status: NOT USABLE
ESMF “bilin”
ne30np4->ne256np4
weights between [0,1]. row sums are perfect. sum error 2.2e-16. conservation error: 3.4e-1 (bilin is not conservative)
mapping file frac_a=0 (BUG?), but frac_b=1ncks --chk_map col sums “frac_a” min/max=0. (perhaps because file’s frac_a=0?)
Status: GOOD
ne120np4->ne512np4
Status: GOOD
ne120np4->ne1024np4
10M out of 120M weights are bad, >> 1.0. max row sum = 5.0
Status: NOT USABLE
ESMF “aave”
We dont consider these maps. We are trying to move away from these maps since they require the “dual grid”, which is difficult to produce and unnatural for GLL grids.
MPAS → np4 grids
The TR improvements suggest we should take another look at “mono” maps for MPAS/EAM coupling. For these maps, testing reported on Transition to TempestRemap for Atmosphere grids recommended using the transpose of the np4->MPAS mono map. So we test the original map here:
TR “mono”
ne256np4 → oRRS18to6 ( with TR from )
weights: good. row sum error: 1e-15 . Conservation error: 4e-14
ne512np4 → oRRS18to6
~4h to compute overlap mesh
..EXCEPTION (src/LinearRemapSE0.cpp, Line 1016) Target grid must be a subset of source grid:
Input mesh area (7.245674993333319e-06) Target area (2.898288183419045e-05)
ne1024np4 → oRRS18to6
MPAS → pg2 grids
We expect SCREAM will ultimately be running on the ne1024pg2 “phys grid”. To create these compsets, we first need to create a domain file which needs the monotone map from the MPAS ocean grid to the pg2 grid.
...
Note2: these maps have a “mask_a,mask_b” and “frac_a,frac_b” variables. It appears “mask” is used to mask out point not used by that component model. For a global atmosphere grid, mask_b=1 for all points. For an MPAS grid, mask_a=1 for all points since the grid only contains ocean points. A POP grid is probably the only example where mask would contain both 0 and 1 values. The MPAS grid has frac_a=1 for all ocean points = all points in the grid. The mapping file value of frac_b ranges from 0..1 (we should check this) on the atmosphere grid, with coastlines given by the factional fractional values.
Note3: for TR FV to FV maps, the --mono options are ignored. monotone conservative maps are constructed using --in_np 1 (piecewise constant reconstruction). higher order maps (--in_np 2) are not monotone.
ncremap built in FV to FV map file generator
oRRS18to6 → ne256pg2 status: GOOD
~3min with 4 threads on Cori frontend node
Weights are good. row sum and conservation error: 6e-13
oRRS18to6 → ne1024pg2
~40 min on Compy with 4 threads. 1.5 h on Cori-Haswell node
Weights are good. row sum error 2e-12, conservation error 1e-12
TR “mono” . (TR from TR from with new --correct_areas option)
oRRS18to6 → ne256pg2
takes about 7h to producestatus: GOOD
2.5h on Anvil (single processor)
Weights are good. Row sums are bad (1.2e-6) Column sum error 1.1e-6
Status: usable for development. NOT USABLE FOR PRODUCTION
oRRS18to6 → ne1024pg2: too expensive/too much memory. need MBTR to compute.
error: 7e-16. Conservation error 7e-16
vortex test field l2/max error: 3.6145e-05 4.3881e-04
oRRS18to6 → ne512pg2: status GOOD
~9h to compute overlap map
weights: good. row sum error: 7e-16. Conservation error 7e-16
vortex: l2, linf: 1.7867e-04 2.2985e-03
oRRS18to6 → ne1024pg2: status: GOOD
15.5h on Anvil, using 20GB
weights: good. row sum error: 4e-16. conservation error: 1e15
MBTR “mono”. Status: NOT USABLE
oRRS18to6 → ne256pg2
Weights are good, but 316 MPAS grid points have weight=0. This is incorrect and corrupts other diagnostics. Status: NOT USABLE
ESMF “aave”
TR mono maps have large loss of precision. can be used for development, but not acceptable long term
- MBTR maps have bugs (missing source points)
oRRS18to6 → ne256pg2. Status: GOOD
weights are good. row sum error 1e-11max 8.5e-12. Conservation error 1.5e-11
Status: GOODvortex test field l2/max error: 3.6144e-05 4.3881e-04
oRRS18to6 → ne512pg2. Status: GOOD
weights are good. row sum error 1e3e-11. conservation error 1e-11
Status: GOODvortex: l2, linf: 1.7866e-04 2.2985e-03
oRRS18to6 → ne1024pg2. Status: NOT USABLE
35min on Cori Haswell node
( /global/project/projectdirs/acme/taylorm/mapping/map_oRRS18to6v3_to_ne1024pg2_aave.nc.BAD )
weights are bad: max 2.6
issue: ncks --chk_map row sum error 3e-11 ( so not seeing the 2.6 weight - it’s probably in a region with frac_a or frac_b << 1)
AnalyzeMap reports rowsum = 3 (consistent with 2.6 weight)
ncks --chk_map conservation error: 3.1e-11
AnalyzeMap:
Status: NOT USABLE
Outstanding issues
, row sum max 3.0. conservation error 4e-4
same issue with ESMF v8.
Outstanding issues
MBTR FV->FV error - missing source points. It appears MBTR maps have a similar loss of precision as TR maps, but this could be corrupted du to the missing source points.
MBTR SE->SE: montone weights are bad, either NaN or >1
ESMF “aave” and “bilin” maps are good up ot to ne512pg2, but both produce too large weights with ne1024pg2
AnalyzeMap
for MPAS->SE map, row sum calculation reports large errors, for points where frac_b<<1. is this due to lack of normalize row sums with frac_b?
AnalyzeMap (Added in TR as of )
request: add min/max of map file frac_a, frac_b data to output
request: check for NaNs when printing weight min/max
ncks --chk_map
add min/max of frac_a, frac_b to output
check of NaNs when printing min/max
for MPAS->atm grid with bad weights (> 1.0) not detected in row sums. Is this due to normalzation of row sums by frac_b?
for non global (MPAS) source grid, prints tens of thousands of useless consistency warnings
(both SE->SE and FV->SE)
...
Command line options used:
New versions required:
TR: code from or later
NCO 4.9.2-alpha05 or newer
Code Block |
---|
#atm grid
set NE=1024
set atmgridnp4 = TEMPEST_NE${NE}.g
set atmgrid = TEMPEST_NE${NE}pg2.g # used by TR, MBTR
set atm_scrip = TEMPEST_NE${NE}pg2.scrip.nc # used by ESMF, ncremap
# option "--alt" now the default and has to be removed from newer TR versions:
GenerateCSMesh --res $NE --out_format Netcdf4 --file $atmgridnp4
GenerateVolumetricMesh --in $atmgridnp4 --out $atmgrid --np 2 --uniform
ConvertExodusToSCRIP --in $atmgrid --out $atm_scrip
# ocean grid:
set target = ocean.oRRS18to6v3.scrip.181106.nc |
Produce conservative monotone mapping:
Code Block |
---|
# NCREMAP:
set mapname = map_oRRS18to6v3_to_ne${NE}pg2_nco.nc
ncremap -6 --thr_nbr=4 --grd_src=$target --grd_dst=$atm_scrip --map=$mapname
ncatted -a grid_file_src,global,a,c,"$target" $mapname
ncatted -a grid_file_dst,global,a,c,"$atm_scrip" $mapname
#ESMF:
set mapname = map_oRRS18to6v3_to_ne${NE}pg2_aave.nc
srun -N 1 -c 4 -C haswell ESMF_RegridWeightGen -s $target -d $atm_scrip -i --src_regional \
--method conserve --netcdf4 -w $mapname
#TR:
set mapname = map_oRRS18to6v3_to_ne${NE}pg2_mono.nc
GenerateOverlapMesh --b $atmgrid --a $target --out overlap.g
GenerateOfflineMap --out_mesh $atmgrid --in_mesh $target --ov_mesh overlap.g \
--in_type fv --in_np 1 --out_type fv --out_double --out_format Netcdf4 \
--correct_areas --out_map $mapname
|
To check maps, use “ncks --chk_map”:
weights: verify that all weights are between [0,1], none of them are NaNs.
row sums: The row sum consistency check measures the ability of the map to preserve the constant function.
ocean->atm maps: max value should be close to 1.0. min value will be between [0..1] at land and fractional land points. Error reported above is from the max value.
atm->ocean maps: min/max value should be close to 1.0 for all rows with nonzero weights (since atmosphere grid is global)
atm->atm maps: min/max values should be close to 1.0
area weighted column sums: this measures if the map is conservative.
ocean->atm maps: area_b weighted column sum should equal area_a for all columns.
atm->ocean maps: area_b weighted column sum should equal area_a at all ocean points, and be between [0,1] at fractional (coastline) points.
atm->atm maps: equal for all columns.
Older versions of ncremap option is missing global attributes needed by CIME’s domain generation code. they can be added via:
ncatted -a grid_file_src,global,a,c,"$target" $mapname
ncatted -a grid_file_dst,global,a,c,"$atm_scrip" $mapname