Versions Compared

Old Version 37

changes.mady.by.user Oksana Guba

Saved on

compared with

New Version 38

changes.mady.by.user Oksana Guba

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • HOMME mesh: A HOMME mesh is either built in HOMME internally, based on Number of Elements (NE) per cube edge (so there is no file that HOMME reads), or an Exodus *.g mesh that lists conforming quadrilaterals. See Atmosphere Grids and the very first plot for an example of a HOMME uniform mesh (quadrilaterals are blue, quadrature points, or degrees of freedom, are green). 
  • SCRIP file: A mesh that is required to couple HOMME to other parts of the model. It is a dual grid for the HOMME grid. It contains spherical polygons that are centered (in a general sense) at HOMME grid (quadrature) points. Its name usually contains suffix _scrip. See Atmosphere Grids for more information. Elaborate on _chevrons, _pentagons..., including information on different options for dual grids, like dual grids constructed with chevrons and pentagons.
  • LATLON file: A file that contains latlon data of HOMME quadrature points/DOFs in a certain format. Its name usually contains suffix _latlon. See Atmosphere Grids for more information. This file is not needed to create domain files.
  • _aave file: A conservative mapping file from one SCRIP file to another with suffix _aave in its name. Needed to construct domain files.
  • _bilin file: A bilinear, not conservative, mapping file from one SCRIP file to another with suffix _bilin in its name.
  • Domain files: Add to this.

...

  1. Action (1) has to be performed by either:

    A) HOMME template run and ncl scripts if the mesh is uniform (NE=4, NE=10, etc.). An example of a template run is in ACME HOMME test suite. The run does not need a mesh file, it only uses NE parameter. Once the template run generates an *.nc file, use it in scripts HOMME2META.ncl and HOMME2SCRIP.ncl to get files *_latlon.nc and *_scrip.nc . The ncl scripts are in ACME/components/homme/test/template/ .

    The only difference between template runs and usual HOMME runs is in which variables are recorded in output *nc files. Output variables are controlled by an input *nl file that HOMME takes in. In the template *nl file in HOMME test suite these vars are listed for output: 

    output_varnames1='phys_area','area','corners','hypervis','cv_lat','cv_lon','phys_lat','phys_lon','phys_cv_lat','phys_cv_lon'

    Note that soon 'phys_area' which is used for FVM transport in HOMME, will be removed from the code.

    Template runs need only 1 time step.

    B) A Matlab script if HOMME mesh is a refined mesh (NE=0). How to create a refined HOMME mesh is not a topic of this page. Refer to SquadGen utility as one of tools. Once a *.g file is created, the Matlab script creates a dual grid in *_scrip.nc file and latlon information in *_latlon.nc . The Matlab script is now checked out to ACME git repository ACME-Climate/PreAndPostProcessingScripts/spectral_elements_grid_utilities . The script that creates both files is dualgridgenerate.m . A script that performs some verification checks on the resulting mesh is on the way geometric_statistics_twisting_center_location.m .

    Note that route A) won’t work with locally refined meshes since HOMME template runs do not handle such grids. Technically, route B) works for uniform meshes if they are recorded in *.g file. However, due to recent changes in HOMME for consistency of elements’ areas, one should be careful to use the same consistency method in HOMME during a simulation as in Matlab when constructing dual meshes. To keep it short, B) is not recommended for uniform meshes. 

    C) Action 1 and 2 can be done by Tempest Remap. Details will be added later.

  2. Action (2) maps HOMME grid from/to another grid (SCRIP file), records mappings in _aave*nc and _bilin*nc files, and is done by an (A) ESMF utility or (B) ncremap, see comment by Charlie Zender below. Most recent scripts for this action are in https://github.com/mt5555/remap-ncl/ , the script is makemap.sh . If, for example, another grid is an ocean grid, then one should add ‘ --src_regional -i ‘ to the command line of the ESMF utility (as said in makemap.sh ) since ocean grids do not cover the whole globe. To create mapping files atmosphere->ocean, one can run a batch script with a command line similar to

    ./remap-ncl/makemap.sh conusx4v1np4b_scrip.nc ocean.RRS.15-5km_scrip_150722.nc map_conusx4v1np4b_to_oRRS15to5

    It takes approx. 20 min on skybridge for these particular files and 32 cores. To change number of cores, modify makemap.job . An attempt to run the script in other direction, ocean→atmosphere did ocean→atmosphere, did not work. It also seems unnecessary because the next step takes only files from atmosphere->ocean iteration.

    By Charlie Zender:

    Action (2) can be performed by ncremap (documentation) which, among other capabilities, uses ESMF or Tempest to generate mapfiles between *_scrip.nc and *.g grids. One benefit of ncremap is the improved provenance retained in the metadata of the generated maps. In theory, the following two ncremap calls could replace makemap.sh:

    ncremap -w esmf -a conserve -s conusx4v1np4bs conusx4v1np4b_scrip.nc nc -g oceang ocean.RRS.15-5km_scrip_150722.nc nc -m mapm map_conusx4v1np4b_to_oRRS15to5_aave.nc
    ncremap -w esmf -a bilinear -s conusx4v1np4bs conusx4v1np4b_scrip.nc nc -g oceang ocean.RRS.15-5km_scrip_150722.nc nc -m mapm map_conusx4v1np4b_to_oRRS15to5_bilin.nc

    (and dash-w tempest without any dash-a option would use Tempest for the maps). However, no one to my knowledge has yet used ncremap for the intermediate step (i.e., step 2) of making domain files (in contrast to end-product mapfiles, which is well-tested). I would be interested in any feedback from those who make domain files on the efficacy of using ncremap.

  3. Action (3) uses gen_domain utility from ACME/CESM. There are instructions to build it in ACME/cime/tools/mapping/gen_domain_files/README (ignore INSTALL, there is a typo). After gen_domain is built, the command is, for example,
    mpiexec -n 1 ~/ACME/cime/tools/mapping/gen_domain_files/gen_domain -m map_oRRS15to5_to_conusx4v1np4b_aave.nc -o oRRS15to5 -l ne0np4_conus_x4v1_lowcon

    For domain files, one should find proper aliases (in our example aliases are oRRS15to5 and ne0np4_conus_x4v1_lowcon) for atmosphere and the other grid in ACME/cime/scripts/Tools/config_grid.xml . The resulting files are

    domain.lnd.ne0np4_conus_x4v1_lowcon_oRRS15to5.160808.nc

    domain.ocn.ne0np4_conus_x4v1_lowcon_oRRS15to5.160808.nc

    domain.ocn.oRRS15to5.160808.nc



More details:

  • Running Matlab can take a few hours (10+). One can turn off plots, insert ‘exit;’ at the end of dualgridgenerate.m and use command
    nohup matlab -nodisplay -nosplash -r dualgridgenerate > out.txt &
  • To build gen_domain on skybridge, file ACME/cime/machines/env_mach_specific.skybridge was modified to match modules and (p)netcdif paths with ACME/components/homme/cmake/machineFiles/skybridge.cmake. It also required module mkl/14.0 to run.
  • Running gen_domain in serial did not work on skybridge (segfault). It helped to allocate a node and run on 1 core with mpiexec.

...