{"type":"doc","content":[{"type":"paragraph","content":[{"text":"The purpose of this page is to document the procedure for adding support for new atmosphere grids. The process should be the same for new uniform resolutions as well as for new regionally-refined meshes, although some settings will need to be changed for new regionally-refined mesh configurations. This page is a work in progress, and will be updated as this process is refined and (eventually) made more automated. This documentation is an update of a document written by ","type":"text"},{"type":"mention","attrs":{"id":"557058:d2854cab-58fe-4b0f-a713-b8aadd0242a1","text":"@Mark Taylor"}},{"text":" and Colin Zarzycki, available as a Google Doc ","type":"text"},{"text":"here","type":"text","marks":[{"type":"link","attrs":{"href":"https://docs.google.com/document/d/1ymlTgKz2SIvveRS72roKvNHN6a79B4TLOGrypPjRvg0/edit?usp=sharing"}}]},{"text":". ","type":"text"}]},{"type":"paragraph","content":[{"text":"Table of Contents","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"2b8b720f-b590-4572-b018-5e04c4a320a1"}},{"type":"underline"},{"type":"strong"}]}]},{"type":"extension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"toc","parameters":{"macroParams":{},"macroMetadata":{"macroId":{"value":"a2fac3ef-d3bc-4a29-9d79-0432380d9015"},"schemaVersion":{"value":"1"},"title":"Table of Contents"}},"localId":"fb3ae397-ec8d-485f-b52f-9a446b80ea25"}},{"type":"paragraph","content":[{"text":"Child Pages","type":"text","marks":[{"type":"underline"},{"type":"strong"}]}]},{"type":"extension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"children","parameters":{"macroParams":{},"macroMetadata":{"macroId":{"value":"24715120-d860-4775-8c4b-d5763dee886c"},"schemaVersion":{"value":"2"},"title":"Child pages (Children Display)"}},"localId":"33da0cd8-0881-4810-8738-52197bc525a1"}},{"type":"paragraph","content":[{"text":"Useful Example Pages:","type":"text","marks":[{"type":"underline"},{"type":"strong"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"type":"inlineCard","attrs":{"url":"https://acme-climate.atlassian.net/wiki/spaces/NGDNA/pages/3086549115"}},{"text":" ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":" ","type":"text"},{"text":"NE512 and NE1024 configurations","type":"text","marks":[{"type":"link","attrs":{"href":"#"}}]}]}]},{"type":"listItem","content":[{"type":"paragraph"}]}]},{"type":"table","attrs":{"layout":"full-width","width":1800.0,"localId":"7413f489-7493-48bd-bbeb-f4c1d9ffd6da"},"content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":[111.0]},"content":[{"type":"paragraph","content":[{"text":"File","type":"text"}]}]},{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":[94.0]},"content":[{"type":"paragraph","content":[{"text":"Tool","type":"text"}]}]},{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":[60.0]},"content":[{"type":"paragraph","content":[{"text":"Type","type":"text"}]}]},{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":[1157.0]},"content":[{"type":"paragraph","content":[{"text":"Note","type":"text"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[111.0]},"content":[{"type":"paragraph","content":[{"type":"hardBreak"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[94.0]},"content":[{"type":"paragraph","content":[{"text":"ncks, ncremap","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[60.0]},"content":[{"type":"paragraph","content":[{"text":"NCO","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[1157.0]},"content":[{"type":"paragraph","content":[{"type":"hardBreak"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[111.0]},"content":[{"type":"paragraph","content":[{"text":"mapping files and mesh template files","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[94.0]},"content":[{"type":"paragraph","content":[{"text":"TempestRemap","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[60.0]},"content":[{"type":"paragraph","content":[{"text":"C++","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[1157.0]},"content":[{"type":"paragraph","content":[{"text":"GenerateCSMesh: make cubed sphere Exodus (.g) files (spectral element \"np\" grid)","type":"text"},{"type":"hardBreak"},{"text":"GenerateVolumetricMesh create a FV \"pg\" from a spectral element \"np\" grid","type":"text"},{"type":"hardBreak"},{"text":"ConvertMeshToSCRIP convert a FV \"pg\" Exodus file into a SCRIP file","type":"text"},{"type":"hardBreak"},{"text":"GenerateOverlapMesh: used in making mapping files","type":"text"},{"type":"hardBreak"},{"text":"GenerateOfflineMap: generate mapping files. Only tool which can make mapping files directly from SE Exodus files. ","type":"text"},{"type":"hardBreak"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[111.0]},"content":[{"type":"paragraph","content":[{"text":"RRM Exodus (.g) mesh files","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[94.0]},"content":[{"type":"paragraph","content":[{"text":"SquadGen","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[60.0]},"content":[{"type":"paragraph","content":[{"text":"C++","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[1157.0]},"content":[{"type":"paragraph","content":[{"type":"hardBreak"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[111.0]},"content":[{"type":"paragraph","content":[{"text":"topo files","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[94.0]},"content":[{"type":"paragraph","content":[{"text":"homme_tool","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[60.0]},"content":[{"type":"paragraph","content":[{"text":"Fortran","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[1157.0]},"content":[{"type":"paragraph","content":[{"text":"Included with E3SM. Should build and run on any system which can run the HOMME_P24.f19_g16_rx1.A test","type":"text"}]},{"type":"paragraph","content":[{"text":"Generate (obsolete) SCRIP files for the spectral element \"np\" dual grid","type":"text"}]},{"type":"paragraph","content":[{"text":"Used for topo smoothing for both FV \"pg\" and SE \"np\" grids. ","type":"text"}]},{"type":"paragraph","content":[{"text":"Can also do parallel interpolation from SE np4 grid to any SCRIP grid","type":"text"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[111.0]},"content":[{"type":"paragraph","content":[{"text":"topo files","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[94.0]},"content":[{"type":"paragraph","content":[{"text":"cube_to_target","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[60.0]},"content":[{"type":"paragraph","content":[{"text":"Fortran","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[1157.0]},"content":[{"type":"paragraph","content":[{"text":"NCAR utility for generating unsmoothed topography from high-res USGF data, and generating surface roughness fields from smoothed topography","type":"text"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[111.0]},"content":[{"type":"paragraph","content":[{"text":"mapping files","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[94.0]},"content":[{"type":"paragraph","content":[{"text":"ESMF_Regridweightgen","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[60.0]},"content":[{"type":"paragraph","content":[{"type":"hardBreak"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[1157.0]},"content":[{"type":"paragraph","content":[{"text":"Make FV->FV mapping files from SCRIP grid template files","type":"text"}]},{"type":"paragraph","content":[{"text":"Only tool which supports the montone 2nd order \"bilin\" map","type":"text"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[111.0]},"content":[{"type":"paragraph","content":[{"type":"hardBreak"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[94.0]},"content":[{"type":"paragraph","content":[{"text":"gen_domain_files","type":"text","marks":[{"type":"code"}]},{"text":" ","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[60.0]},"content":[{"type":"paragraph","content":[{"text":"CIME and ELM tools","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[1157.0]},"content":[{"type":"paragraph","content":[{"type":"hardBreak"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[111.0]},"content":[{"type":"paragraph","content":[{"text":"land surface dataset ","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[94.0]},"content":[{"type":"paragraph","content":[{"text":"mksurfdata.pl, mksurfdata_map,","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[60.0]},"content":[{"type":"paragraph","content":[{"text":"Perl and Fortran","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[1157.0]},"content":[{"type":"paragraph","content":[{"type":"hardBreak"}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[111.0]},"content":[{"type":"paragraph","content":[{"text":"ELM initial condition","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[94.0]},"content":[{"type":"paragraph","content":[{"text":"interpinic","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[60.0]},"content":[{"type":"paragraph","content":[{"text":"Fortran","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[1157.0]},"content":[{"type":"paragraph","content":[{"text":"4 options:","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"cold start: (no IC file). only suitable for testing functionality in other components.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Run long spinup with prescribed atmosphere ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Interpolate from a spunup IC file from a different simulation, via \"interpinc\" utility","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Inline interpolation for land initial condition is available in E3SM code, but this capability might get broken with the new land subgrid structure","type":"text"}]}]}]},{"type":"paragraph","content":[{"type":"hardBreak"}]},{"type":"paragraph","content":[{"type":"hardBreak"}]}]}]}]},{"type":"paragraph","content":[{"text":"Archives of already-created atmosphere grid and mapping files can be accessed at ","type":"text"},{"text":"https://web.lcrc.anl.gov/public/e3sm/mapping/grids/","type":"text","marks":[{"type":"link","attrs":{"href":"https://web.lcrc.anl.gov/public/e3sm/mapping/grids/"}}]},{"text":" and ","type":"text"},{"text":"https://web.lcrc.anl.gov/public/e3sm/mapping/maps/","type":"text","marks":[{"type":"link","attrs":{"href":"https://web.lcrc.anl.gov/public/e3sm/mapping/maps/"}}]},{"text":" . This page focuses on creating new versions of these files.","type":"text"}]},{"type":"paragraph","content":[{"text":"For the purpose of this step-by-step guide, we will walk through each step using an example of regenerating the ne4 grid that is already currently supported. For a new grid (i.e., ne512), just replace ne4 with the desired grid identifier. This is determined in Step 1 Generate a new \"grid\" file below, and the rest should follow.","type":"text"}]},{"type":"paragraph","content":[{"text":"For this example, I will assume a few environment variables, which we should set now. First, determine where to put the generated output files. For big grids, this can take up quite a bit of space, so choose a location with sufficient space. I have a symlink to my CSCRATCH directory on NESRC in my home directory, so I will put grid files there for this example:","type":"text"}]},{"type":"codeBlock","content":[{"text":"output_root=${HOME}/cscratch/e3sm/grids/ne4\nmkdir -p ${output_root}","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Types of Atmosphere grid metadata files","type":"text"}]},{"type":"paragraph","content":[{"text":"See ","type":"text"},{"text":"SE Atmosphere Grid Overview (EAM & CAM)","type":"text","marks":[{"type":"link","attrs":{"href":"https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/34113147"}}]},{"text":" for description of the spectral elements, GLL nodes, subcell grid and dual grid. ","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Exodus","type":"text","marks":[{"type":"strong"}]},{"text":" file: \"ne4.g\". This is a netcdf file following ","type":"text"},{"text":"Exodus","type":"text","marks":[{"type":"link","attrs":{"href":"https://gsjaardema.github.io/seacas/html/exodus_formats.html"}}]},{"text":" ","type":"text"},{"text":"conventions","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"cad64db5-0ce8-448e-a4b7-7ae05c5cd0bc"}}]},{"text":". It gives the corners of all elements on the sphere and their connectivity. It is independent of the polynomial order used inside the element (\"np\"). ","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"This file is used by TempestRemap (TR) to generate mapping files. The polynomial order is a command line option and the GLL nodes are internally generated by TR. ","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"SCRIP","type":"text","marks":[{"type":"strong"}]},{"text":" file: \"ne4pg2.scrip.nc\". This file contains a description of the atmosphere physics grid n the format used by the original incremental remap tool SCRIP. It is used for most output and also used to generate mapping files between components and for post-processing of most output.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Less common “GLL” metadata files needed for specialized purposes:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"SCRIP","type":"text","marks":[{"type":"strong"}]},{"text":" file: \"ne4np4_scrip.nc\". This file contains a description (SCRIP format) of the ","type":"text"},{"text":"GLL dual grid","type":"text","marks":[{"type":"em"}]},{"text":". It includes the locations of the GLL nodes and artificial bounding polygons around those nodes. Ideally the spherical area of each polygon will match the GLL weight (\"exact GLL areas\"), but not all tools can achieve exact areas. Inexact areas ","type":"text"},{"text":"does not","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"e72e3ebd-524d-440b-839b-6a3047f0239e"}}]},{"text":" impact the accuracy of the resulting mapping algorithm, it just means that mass will not be exactly conserved by the mapping algorithm. ","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Needed in E3SMv2 configurations by the cube_to_target topography dataset generation tool.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Historically used by E3SMv1 to create ocean/atm mapping files (exact GLL areas are important here, to conserve fluxes)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Historically used by E3SMv1 for regional and sub-gridscale regridding post-processing tools ( ","type":"text"},{"text":"https://acme-climate.atlassian.net/wiki/spaces/SIM/pages/754286611/Regridding+E3SM+Data+with+ncremap","type":"text","marks":[{"type":"link","attrs":{"href":"https://acme-climate.atlassian.net/wiki/spaces/SIM/pages/754286611/Regridding+E3SM+Data+with+ncremap"}}]},{"text":", ","type":"text"},{"text":"http://nco.sf.net/nco.html#sgs","type":"text","marks":[{"type":"link","attrs":{"href":"http://nco.sf.net/nco.html#sgs"}}]},{"text":")","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"latlon","type":"text","marks":[{"type":"strong"}]},{"text":" file: \"ne4np4_latlon.nc\". This file contains a list of all the GLL nodes in the mesh (in latitude/longitude coordinates). The list of GLL nodes must be in the the internal HOMME global id ordering, matching the ordering used in CAM and EAM native grid output. It also contains the connectivity of the ","type":"text"},{"text":"GLL subcell grid","type":"text","marks":[{"type":"em"}]},{"text":". ","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"This file is used by CAM's interpic_new utility, and graphics programs Paraview and Visit when plotting native grid GLL output.","type":"text"}]}]}]}]}]}]}]},{"type":"paragraph","content":[{"type":"hardBreak"}]},{"type":"rule"},{"type":"heading","attrs":{"level":1},"content":[{"text":"Step-by-step guide","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"1. Generate a new atmosphere \"grid\" file","type":"text"}]},{"type":"paragraph","content":[{"text":"Requirements: ","type":"text"},{"text":"TempestRemap","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/ClimateGlobalChange/tempestremap"}}]}]},{"type":"paragraph","content":[{"text":"In order to generate mapping files to do interpolation between a new atmosphere grid and the other component grids, we need to create a file that describes the atmosphere grid representation. The TempestRemap tool (","type":"text"},{"text":"https://github.com/ClimateGlobalChange/tempestremap","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/ClimateGlobalChange/tempestremap"}}]},{"text":") can be used to recreate the internal representation of the spectral element grids used in HOMME. The grid mesh file will be saved in an \"exodus\" type file (with a .g extension). ","type":"text"}]},{"type":"paragraph","content":[{"text":"TempestRemap can now be pulled in with NCO via conda, or can be built from source from the Github repository. To build from source, make sure a netCDF module is loaded. This can be accomplished by sourcing an ","type":"text"},{"text":"env_mach_specific.sh","type":"text","marks":[{"type":"code"}]},{"text":" from a working case on the target machine before building. Then, ","type":"text"}]},{"type":"codeBlock","content":[{"text":"git clone https://github.com/ClimateGlobalChange/tempestremap.git\ncd tempestremap\nmake -f Makefile.gmake\ncd ..","type":"text"}]},{"type":"paragraph","content":[{"text":"This will build TempestRemap into tempestremap/bin. For the rest of the tutorial, assume that the environment variable ","type":"text"},{"text":"${tempest_root}","type":"text","marks":[{"type":"code"}]},{"text":" points to this directory (or where a conda-build of TempestRemap exists):","type":"text"}]},{"type":"codeBlock","content":[{"text":"tempest_root=${PWD}/tempestremap","type":"text"}]},{"type":"paragraph","content":[{"text":"We can generate an ne4 mesh easily now:","type":"text"}]},{"type":"codeBlock","content":[{"text":"${tempest_root}/bin/GenerateCSMesh --alt --res 4 --file ${output_root}/ne4.g","type":"text"}]},{"type":"paragraph","content":[{"text":"For RRM grids, SQuadGen should be used (available in a separate repository on Github ","type":"text"},{"text":"here","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/ClimateGlobalChange/squadgen"}}]},{"text":"). See ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/1028128773"}},{"text":" for details and examples. The example above follows the convention ","type":"text"},{"text":"ne.g","type":"text","marks":[{"type":"code"}]},{"text":". RRM meshes should follow the convention ","type":"text"},{"text":"_x.g","type":"text","marks":[{"type":"code"}]},{"text":". For example, for a RRM with 4x refinement from ne30 to ne120 over CONUS, we should use the convention ","type":"text"},{"text":"conus_ne30x4.g","type":"text","marks":[{"type":"code"}]},{"text":" (note that existing meshes use the naming convention ","type":"text"},{"text":"xv.g","type":"text","marks":[{"type":"code"}]},{"text":", but future meshes should use the new naming convention; we also plan to eventually change the extension to ","type":"text"},{"text":"_exodus.nc","type":"text","marks":[{"type":"code"}]},{"text":", but I believe this will break ","type":"text"},{"text":"nco","type":"text","marks":[{"type":"code"}]},{"text":" functionality to automatically detect when to use ","type":"text"},{"text":"TempestRemap","type":"text","marks":[{"type":"code"}]},{"text":" for mapping when detecting exodus files).","type":"text"}]},{"type":"paragraph","content":[{"text":"The Exodus file contains only information about the position of the spectral element on the sphere. For SE aware utilities such as TempestRemap, they can use the polynomial order and the reference element map to fill in necessary data such as the locations of the nodal GLL points. For non-SE aware utilities, we need additional meta data, described in the next section. ","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"2A. Generate control volume mesh files for E3SM v2 \"pg2\" grids ","type":"text"}]},{"type":"paragraph","content":[{"text":"Requirements:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"exodus mesh file","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"TempestRemap","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"In E3SM v2, we switched to running physics on a FV pg2 grid and mapping files between the atmosphere and other components will be FV to FV type maps using the pg2 grids. These can be generated by TempestRemap directly from the exodus mesh file:","type":"text"}]},{"type":"codeBlock","content":[{"text":"${tempest_root}/bin/GenerateVolumetricMesh --in ne4.g --out ne4pg2.g --np 2 --uniform \n${tempest_root}/bin/ConvertMeshToSCRIP --in ne4pg2.g --out ne4pg2.scrip.nc ","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"2B. Generate \"dual grid\" mesh files (SCRIP and lat/lon format) for E3SM v1 \"np4\" GLL grids","type":"text"}]},{"type":"paragraph","content":[{"text":"Requirements:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"exodus mesh file","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"homme_tool","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Note: in E3SM v2, we switched to running physics on a FV pg2 grid and mapping files between the atmosphere and other components will be FV to FV type maps using the pg2 grids. The spectral element \"np4\" grid is still used internally by the dynamics and for initial conditions, so the metadata described in this section is still needed for some analysis and preparation of initial conditions. ","type":"text"}]},{"type":"paragraph","content":[{"text":"Mapping files used by the coupler for mapping fluxes between SE \"np4\" and FV grids should be generated with TempestRemap and only need the exodus grid description (which provides the locations of the corners of the quadrilaterals that form the elements of the cube-sphere mesh) generated in Step 1 above. However, a handful of pre- and post-processing tools require a finite-volume equivalent of the spectral element grid (these tools include the surface roughness calculation in the land tool ","type":"text"},{"text":"cube_to_target","type":"text","marks":[{"type":"code"}]},{"text":", ESMF mapping files used for interpolating land surface data to target grid, and post-processing regional and subgrid remapping tools). We refer to this finite-volume description of the SE grid as the \"dual grid\" to the SE grid (see page describing atmosphere grids in more detail ","type":"text"},{"text":"here","type":"text","marks":[{"type":"link","attrs":{"href":"https://acme-climate.atlassian.net/wiki/spaces/Docs/pages/34113147/Atmosphere+Grids"}}]},{"text":"). ","type":"text"}]},{"type":"paragraph","content":[{"text":"Generate SCRIP “dual grid” with homme_tool: ","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Be sure your environment matches the software environment loaded by E3SM by executing the output of this command: e3sm/cime/scripts/Tools/get_case_env","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Use cmake to configure and compile standalone HOMME. On a supported platform with the CIME environement, this should work out-of-the-box. See e3sm/components/homme/README.cmake","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"compile the HOMME tool utility: ","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"cd /path/to/workingdir ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"make -j4 homme_tool","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"executable: /path/to/workingdir/src/tool/homme_tool","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Edit e3sm/components/homme/test/tool/namelist/template.nl and specify the grid resolution or RRM file","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"d19a5575-5fb9-4d11-87fa-fa14658f1ed4"}}]}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"For ne512, this would be ","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"d19a5575-5fb9-4d11-87fa-fa14658f1ed4"}}]},{"text":"ne = 512","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"d19a5575-5fb9-4d11-87fa-fa14658f1ed4"}},{"type":"code"}]},{"text":". For RRM grids, leave ","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"d19a5575-5fb9-4d11-87fa-fa14658f1ed4"}}]},{"text":"ne = 0","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"d19a5575-5fb9-4d11-87fa-fa14658f1ed4"}},{"type":"code"}]},{"text":", but will need to edit where the exodus grid file comes from","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"d19a5575-5fb9-4d11-87fa-fa14658f1ed4"}}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"for non-RRM grids using the older E3SM v1 dycore, add cubed_sphere_map=0 to template.nl","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"d19a5575-5fb9-4d11-87fa-fa14658f1ed4"}}]}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"See e3sm/components/homme/test/tool/test.job for examples of how to run the script and then use an NCL utilities to process the tool output into SCRIP and latlon formats. ","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Specific details for running at NERSC on Cori(knl):","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"5c11ea04-8109-4075-86fd-04671fe92f50"}}]}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Create a batch script hange \"account\" in the sbatch directives at the top of the script. For example, set ","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"5c11ea04-8109-4075-86fd-04671fe92f50"}}]},{"text":"#SBATCH --account=e3sm","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"5c11ea04-8109-4075-86fd-04671fe92f50"}},{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"cmake -C /path/to/e3sm/components/homme/cmake/machineFiles/cori-knl.cmake -DPREQX_NP=4 /path/to/workingdir","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Make sure a working NCL is in your PATH. On Cori, add the following to the script: ","type":"text"},{"text":"module load ncl","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"2C. Atmospheric mesh quality","type":"text"}]},{"type":"paragraph","content":[{"text":"Atmospheric RRM mesh quality can be measured with the “Max Dinv-based element distortion” metric. This will be printed in the log file for standalone HOMME or EAM simulations (and can be obtained from the log files during the topo generation step). It measures how distorted the elements become in the mesh transition region. It is the ratio of the two singular values of the 2x2 derivative matrix of the element map to the unit square, representing the ration of the largest length scale to the smallest length scale. A grid of perfect quadrilaterals will have a value of 1.0. The equal-angle cubed-sphere grid has a value of 1.7. A high quality regionally refined grid will have a value less than 4. With a high quality grid, usually one can run with the timesteps used in a uniform grid with matching fine resolution. RRM grids with a value > 4 may require smaller timesteps for stability. Very large values indicate a problem with the grid and it should be redesigned. ","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"3. Generate mapping files","type":"text"}]},{"type":"paragraph","content":[{"text":"Requirements:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"TempestRemap","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"cba0a1e8-8a4f-4fc8-baa9-242657aa046f"}},{"type":"link","attrs":{"href":"https://github.com/ClimateGlobalChange/tempestremap"}}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"ESMF_RegridWeightGen","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"ncremap","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"grid descriptor files for each component that exists on a different grid ","type":"text"},{"type":"hardBreak"},{"text":"(atmosphere, ocean, possibly land if on a different grid than the atmosphere)","type":"text","marks":[{"type":"em"}]}]}]}]},{"type":"paragraph","content":[{"text":"In order to pass data between different components at runtime, a set of mapping files between each component is generated offline. These mapping files will also be used in Step 4 below (generating domain files).","type":"text"}]},{"type":"paragraph","content":[{"text":"See ","type":"text"},{"text":"Transition to TempestRemap for Atmosphere grids","type":"text","marks":[{"type":"link","attrs":{"href":"https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/178848194"}}]},{"text":" for a discussion of different remap algorithms and when to use each.","type":"text"}]},{"type":"paragraph","content":[{"text":"TempestRemap","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/ClimateGlobalChange/tempestremap"}}]},{"text":" and ESMF are the backends that generate the mapping weights, but this is all nicely encapsulated using ncremap. Tempest is the preferred method for creating mapping files. The ncremap function, calling TempestRemap or ESMF, will decide which of these two tools to use based on the atmospheric input file. If the *.scrip file was used, then ESMF will be called. If the *.g file was used, then TempestRemap will be called. The ESMF tools are adequate for making atmosphere-only-type component sets for E3SM, but this tool is less conservative than TempestRemap. If you are making grids for a coupled run, then TempestRemap should be used. ","type":"text"}]},{"type":"paragraph","content":[{"text":"The easiest way to make sure ncremap is up to date and will work for this workflow, we can source the ESMF-Unified conda environment. ","type":"text"},{"text":"Instructions for supported machines can be found here","type":"text","marks":[{"type":"link","attrs":{"href":"https://acme-climate.atlassian.net/wiki/spaces/EIDMG/pages/780271950/Analysis+Machines+and+Software"}}]},{"text":". For Perlmutter, it is as simple ","type":"text"},{"text":"as:","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"941e9be3-36a9-45b8-aac6-bbd153a4c662"}}]}]},{"type":"codeBlock","content":[{"text":"source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh","type":"text"}]},{"type":"paragraph","content":[{"text":"For unsupported machines you may need to install the conda environment via:","type":"text"}]},{"type":"codeBlock","content":[{"text":"conda install -c conda-forge -c e3sm e3sm-unified","type":"text"}]},{"type":"paragraph","content":[{"text":"TempestRemap was built in Step #1 above. We need to tell ncremap where to find our TempestRemap binaries, so we need to append to our PATH:","type":"text"}]},{"type":"codeBlock","content":[{"text":"PATH=${tempest_path}/bin:${PATH}","type":"text"}]},{"type":"paragraph","content":[{"text":"And now we can use ncremap to generate ALL the needed mapping files between two grids, in ","type":"text"},{"text":"this example","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"e0315c9b-7710-49c8-aece-9036db8b9670"}}]},{"text":" the ne4 atmosphere and the oQU240 ","type":"text"},{"text":"ocean grid","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"6c9752ff-ca0a-48d6-882b-2827be121d25"}}]},{"text":" (for the moment, we will put the land on the same grid as the atmosphere):","type":"text"}]},{"type":"codeBlock","content":[{"text":"atm_grid_file=ne4.g\nocn_grid_file=/global/cfs/cdirs/e3sm/inputdata/cpl/gridmaps/oQU240/ocean.QU.240km.scrip.151209.nc\ncd ${output_root} && ncremap -P mwf -s $ocn_grid_file -g $atm_grid_file --nm_src=oQU240 --nm_dst=ne4np4 --dt_sng=20181114","type":"text"}]},{"type":"paragraph","content":[{"text":"In the above code, the ","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"6dccb4a7-5367-4f18-a1ec-6a73d2f91ea0"}}]},{"text":"\"-P mwf\" option triggers a specific procedure that invokes multiple ncremap commands which each invoke multiple Tempest commands.It produces 3 mapping files each for a2o and o2a. This multitude of mapping files for each component→component case are needed because fluxes need conservative mapping and states need non-conservative mapping. Because \"-P mwf\" results in a great deal of output being flushed to screen, it is mostly suppressed by default. To see all the output or to figure out how to run all the nco and tempest remap commands one-by-one, add \"","type":"text"},{"text":"--dbg_lvl=2","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]},{"text":"\" to the ncremap command. This will print the commands but not execute them. The user can then use one of the printed commands, again with \"","type":"text"},{"text":"--dbg_lvl=2\"","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]},{"text":" to see the actual commands being sent to Tempest.","type":"text"}]},{"type":"paragraph","content":[{"text":"If this command is successful, it should produce many mapping files in ","type":"text"},{"text":"${output_root}","type":"text","marks":[{"type":"code"}]},{"text":", that look something like","type":"text"}]},{"type":"codeBlock","content":[{"text":"map_${atm_grid}_to_${ocn_grid}_${method}.${datestring}.nc\nmap_${ocn_grid}_to_${atm_grid}_${method}.${datestring}.nc\nmap_${atm_grid}_to_${lnd_grid}_${method}.${datestring}.nc\nmap_${lnd_grid}_to_${atm_grid}_${method}.${datestring}.nc\nmap_${lnd_grid}_to_${ocn_grid}_${method}.${datestring}.nc\nmap_${ocn_grid}_to_${lnd_grid}_${method}.${datestring}.nc","type":"text"}]},{"type":"paragraph","content":[{"text":"where ","type":"text"},{"text":"${ocn_grid}, ${atm_grid}, and ${lnd_grid} ","type":"text","marks":[{"type":"code"}]},{"text":"are the names of the ocean and atmos grid provided above in the ","type":"text"},{"text":"--nm_src","type":"text","marks":[{"type":"code"}]},{"text":" and ","type":"text"},{"text":"--nm_dst","type":"text","marks":[{"type":"code"}]},{"text":" arguments, and ","type":"text"},{"text":"${datestring}","type":"text","marks":[{"type":"code"}]},{"text":" is the date string provided above in the ","type":"text"},{"text":"--dt_sng","type":"text","marks":[{"type":"code"}]},{"text":" argument to ","type":"text"},{"text":"ncremap","type":"text","marks":[{"type":"code"}]},{"text":". The ${method} can be monotr, highorder, mono, intbilin, aave, blin, ndtos, nstod, and patc. (TODO: What do all these mean?) If using the tri-grid option, the land grid files will be created from the ${lnd_grd}. ","type":"text"}]},{"type":"paragraph","content":[{"text":"For pg2 grids, the ncremap invocations are as follows, using ne30pg2 atmosphere, oEC60to30v3 ocean, and half-degree land grids as examples.","type":"text"}]},{"type":"codeBlock","content":[{"text":"atm_grid_file=ne30pg2.g\natm_scrip_grid_file=ne30pg2_scrip.nc\nocn_grid_file=ocean.oEC60to30v3.scrip.181106.nc\nlnd_grid_file=SCRIPgrid_0.5x0.5_nomask_c110308.nc\n\natm_name=ne30pg2\nocn_name=oEC60to30v3\nlnd_name=r05\n\n## Conservative, monotone maps.\n\nalg_name=mono\n\ndate=200110\n\nfunction run {\n echo \"src $src dst $dst map $map\"\n ncremap -a tempest --src_grd=$src --dst_grd=$dst -m $map \\\n -W '--in_type fv --in_np 1 --out_type fv --out_np 1 --out_format Classic --correct_areas' \\\n $extra\n}\n\nextra=\"\"\n\nsrc=$ocn_grid_file\ndst=$atm_grid_file\nmap=\"map_${ocn_name}_to_${atm_name}_${alg_name}.${date}.nc\"\nrun\n\nsrc=$atm_grid_file\ndst=$ocn_grid_file\nmap=\"map_${atm_name}_to_${ocn_name}_${alg_name}.${date}.nc\"\nextra=--a2o\nrun\nextra=\"\"\n\nsrc=$lnd_grid_file\ndst=$atm_grid_file\nmap=\"map_${lnd_name}_to_${atm_name}_${alg_name}.${date}.nc\"\nrun\n\nsrc=$atm_grid_file\ndst=$lnd_grid_file\nmap=\"map_${atm_name}_to_${lnd_name}_${alg_name}.${date}.nc\"\nrun\n\n## Nonconservative, monotone maps.\n\nalg_name=bilin\n\nsrc=$atm_scrip_grid_file\ndst=$lnd_grid_file\nmap=\"map_${atm_name}_to_${lnd_name}_${alg_name}.${date}.nc\"\nncremap -a bilinear -s $src -g $dst -m $map -W '--extrap_method nearestidavg'\n\nsrc=$atm_scrip_grid_file\ndst=$ocn_grid_file\nmap=\"map_${atm_name}_to_${ocn_name}_${alg_name}.${date}.nc\"\nncremap -a bilinear -s $src -g $dst -m $map -W '--extrap_method nearestidavg'","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"4. Generate domain files","type":"text"}]},{"type":"paragraph","content":[{"text":"Domain files are needed by the coupler and the land model at runtime. The land model uses the mask to determine where to run and the coupler use the land fraction to merge fluxes from multiple surface types to the atmosphere above them. Domain files are created from the mapping files created in the previous step, using a tool provided with CIME in ","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"6737a05d-3454-4be1-97de-b5dc8b096405"}}]},{"text":"${e3sm_root}/cime/tools/mapping/gen_domain_files","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"6737a05d-3454-4be1-97de-b5dc8b096405"}},{"type":"code"}]},{"text":". This directory contains the source code for the tool (in Fortran 90) and a Makefile. Cloning E3SM is now required to obtain code within the distribution. To clone E3SM, ","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"6737a05d-3454-4be1-97de-b5dc8b096405"}}]}]},{"type":"codeBlock","content":[{"text":"# Checkout E3SM code\ngit clone git@github.com:E3SM-Project/E3SM.git","type":"text"}]},{"type":"paragraph","content":[{"text":"Assuming ","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"6737a05d-3454-4be1-97de-b5dc8b096405"}}]},{"text":"${e3sm_root}","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"6737a05d-3454-4be1-97de-b5dc8b096405"}},{"type":"code"}]},{"text":" is set and the location of TempestRemap binaries is added to ","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"6737a05d-3454-4be1-97de-b5dc8b096405"}}]},{"text":"${PATH}","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"6737a05d-3454-4be1-97de-b5dc8b096405"}},{"type":"code"}]},{"text":", the ","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"6737a05d-3454-4be1-97de-b5dc8b096405"}}]},{"text":"following script","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"6737a05d-3454-4be1-97de-b5dc8b096405"}},{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"e3de523d-cee8-4c55-a601-34fb25614f36"}}]},{"text":" should generate the appropriate domain files:","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"6737a05d-3454-4be1-97de-b5dc8b096405"}}]}]},{"type":"codeBlock","content":[{"text":"#!/bin/bash\n\n# Setup environment\nsource /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh\ne3sm_root=${HOME}/codes/e3sm/branches/master\n\n# Build gen_domain tool\ngen_domain=${e3sm_root}/cime/tools/mapping/gen_domain_files/gen_domain\ncd `dirname ${gen_domain}`/src\neval $(${e3sm_root}/cime/CIME/Tools/get_case_env)\n${e3sm_root}/cime/CIME/scripts/configure --macros-format Makefile --mpilib mpi-serial\ngmake\n\n# Set paths to mapping files\nmapping_root=\"/global/homes/b/bhillma/cscratch/e3sm/grids/ne4\"\nocn_grid_name=oQU240\natm_grid_name=ne4np4\nlnd_grid_name=${atm_grid_name}\n\n# run domain generation tool (from output directory)\ndomain_root=${mapping_root} \nmkdir -p ${domain_root} && cd ${domain_root}\nfor target_grid_name in ${lnd_grid_name} ${atm_grid_name}; do\n \n # Find conservative mapping files, use the latest file generated \n map_ocn_to_target=`ls ${mapping_root}/map_${ocn_grid_name}_to_${target_grid_name}_monotr.*.nc | tail -n1`\n\n # Run domain tool code\n ${gen_domain} -m ${map_ocn_to_target} -o ${ocn_grid_name} -l ${target_grid_name}\n\ndone\n","type":"text"}]},{"type":"paragraph","content":[{"text":"NOTE - on Perlmutter the “OS” environment variable is not set, so to work around this simply set “OS=LINUX“","type":"text"},{"type":"hardBreak"},{"text":"If this command is successful, it should produce many domain files in ","type":"text"},{"text":"${output_root}/domain_file ","type":"text","marks":[{"type":"code"}]},{"text":"that look something like","type":"text"}]},{"type":"codeBlock","content":[{"text":"domain.lnd.${atm_grid}_${ocn_grid}.${datestring}.nc\ndomain.ocn.${atm_grid}_${ocn_grid}.${datestring}.nc\ndomain.ocn.${ocn_grid}.${datestring}.nc\ndomain.lnd.${lnd_grid}_${ocn_grid}.${datestring}.nc\ndomain.ocn.${lnd_grid}_${ocn_grid}.${datestring}.nc","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"5. Generate topography file ","type":"text"}]},{"type":"paragraph","content":[{"text":"Generating the topography and related surface roughness data sets is a detailed process that has been moved to it’s own page, with detailed instructions depending on the model version (V1, V2, V3)","type":"text"}]},{"type":"paragraph","content":[{"type":"inlineCard","attrs":{"url":"https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/2720202817"}},{"text":" ","type":"text"}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":" ","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"6. Generate and spin-up a new atmosphere initial condition","type":"text"}]},{"type":"paragraph","content":[{"text":"Generating a new initial condition for the atmosphere is a two-step process. First, an existing initial condition is interpolated to the target resolution, then the interpolated initial condition is used to spin-up a new initial condition that is in balance and consistent with the dynamics (I am probably not explaining this very well; bottom line, without this step the model will probably blow up if you try to run with the interpolated initial condition, at least for RRM grids). The spin-up requires initially lowering the timestep and increasing the hyperviscosity, and then gradually relaxing these back to more reasonable values. Both of these steps are described below.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Step 1: Generating a \"first-guess\" initial condition","type":"text"}]},{"type":"paragraph","content":[{"text":"A starting point for a new initial condition is first interpolated from an existing initial condition. Traditionally, this has been done using the ","type":"text"},{"text":"interpic_new","type":"text","marks":[{"type":"code"}]},{"text":" tool, which does both horizontal (for new grids) and vertical (for potentially different vertical grids/numbers of levels) interpolation of the fields in the initial condition file. This is a Fortran code that is included in E3SM, within the atmosphere tools directory. Unfortunately, the tool only supports interpolating from lat/lon grids, and cannot interpolate from unstructured to unstructured. So, if you want to use this tool to interpolate an existing initial condition to an SE grid, you will have to start with an older initial condition on an FV grid or similar. Thus, the use of ","type":"text"},{"text":"interpic_new","type":"text","marks":[{"type":"code"}]},{"text":" is NO LONGER SUPPORTED OR RECOMMENDED. The script below for building and using ","type":"text"},{"text":"interpic_new","type":"text","marks":[{"type":"code"}]},{"text":" is included only to document the process in case someone wanted to revive this workflow. The script will not work on NERSC, as the paths do not exists:","type":"text"}]},{"type":"codeBlock","content":[{"text":"#!/bin/bash\n\n# Get machine-specific modules\nenv_mach_specific=env_mach_specific.sh\nsource ${env_mach_specific}\n\ne3sm_root=/home/bhillma/codes/e3sm/branches/master\n\ninterp_root=${e3sm_root}/components/eam/tools/interpic_new\ntemplate_file=${interp_root}/template.nc\n\ninput_atm_ic_file=/projects/ccsm/inputdata/atm/cam/inic/fv/cami-mam3_0000-01-01_0.9x1.25_L30_c100618.nc\nnlevels=72\natm_latlon_file=/gscratch/bhillma/e3sm/grids/conusx8v1/conusx8v1np4b_latlon_170526.nc\n\n# specify a file to pull information about vertical levels from\nif [ ${nlevels} -eq 72 ]; then\n vertical_file=/projects/ccsm/inputdata/atm/cam/inic/homme/cami_mam3_Linoz_0000-01-ne120np4_L72_c160318.nc\nelif [ ${nlevels} -eq 30 ]; then\n vertical_file=/projects/ccsm/inputdata/atm/cam/inic/homme/cami-mam3_0000-01-ne120np4_L30_c110928.nc\nelse\n echo \"No input specified for nlevels=${nlevels}.\"\n exit 1\nfi\n\ndatestring=`date +'%y%m%d'`\ngrid_name=\"conusx8v1\"\noutput_root=\"/gscratch/bhillma/e3sm/grids/${grid_name}\"\noutput_atm_ic_file=${output_root}/cami-mam3_0000-01-${grid_name}_L${nlevels}_c${datestring}.nc\n\n# copy horizontal coordinates to template file\n/projects/ccsm/nco/bin/ncks -O -v lat,lon ${atm_latlon_file} ${template_file}\n\n# copy vertical coordinates from existing initial condition to template file\n/projects/ccsm/nco/bin/ncks -A -v hyai,hybi,hyam,hybm ${vertical_file} ${template_file}\n\n# build tool\ncd ${interp_root}\nmake clean\ngmake\n\n# run the interpolation code\n./interpic -t ${template_file} ${input_atm_ic_file} ${output_atm_ic_file}\n\n# if input initial condition was from an FV grid, rename US->U, VS->V\n# this is hard-coded above, so yes we need to rename US and VS\n/projects/ccsm/nco/bin/ncrename -O -v US,U -v VS,V ${output_atm_ic_file}\n\n# update configuration file\nif [ $? -eq 0 ]; then\n echo \"Successfully created ${output_atm_ic_file}.\"\nelse\n echo \"Something went wrong.\"\n exit 1\nfi","type":"text"}]},{"type":"paragraph","content":[{"text":"A more flexible approach is to use TempestRemap to do the horizontal interpolation from an existing SE (or FV) grid. This is straightforward. For example, to interpolate from an existing ne120 grid:","type":"text"}]},{"type":"codeBlock","content":[{"text":"#!/bin/bash\n \n# Add tempest to path\nexport PATH=tempestremap/bin:${PATH}\n \n# Generate source mesh\nsource_mesh_file=ne120.g\nrm -f ${source_mesh_file}\nGenerateCSMesh --alt --res 120 --file ${source_mesh_file}\n \n# Generate overlap mesh\nGenerateOverlapMesh --a ${source_mesh_file} --b ${atm_mesh_file} --out overlap_mesh.nc\n \n# Generate mapping weights\nGenerateOfflineMap \\\n --in_mesh ${source_mesh_file} --out_mesh ${atm_mesh_file} --ov_mesh overlap_mesh.nc \\\n --in_np 4 --out_np 4 --in_type cgll --out_type cgll --out_map mapping_weights.nc\n \n# Apply mapping weights\nsource activate python3\nsource_initial_condition=\"${inputdata_root}/atm/cam/inic/homme/cami_mam3_Linoz_0000-01-ne120np4_L72_c160318.nc\"\natm_initial_condition=\"${output_root}/initial_conditions/cami_0000-01_${resolution}_L72_c190218.nc\"\nmkdir -p `dirname ${atm_initial_condition}`\nncremap \\\n -4 -m mapping_weights.nc \\\n ${source_initial_condition} ${atm_initial_condition}","type":"text"}]},{"type":"paragraph","content":[{"text":"Note that although this approach is flexible in the source grid, it does require an existing initial condition with the same vertical grid. However, NCO will now do vertical interpolation, and will also wrap TempestRemap horizontal remapping","type":"text"},{"text":". This is well documented on the NCO homepage (","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"e4a7ff9b-e0ab-489f-acf2-d96dca06fdfc"}}]},{"text":"http://nco.sourceforge.net/nco.html","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"e4a7ff9b-e0ab-489f-acf2-d96dca06fdfc"}},{"type":"link","attrs":{"href":"http://nco.sourceforge.net/nco.html"}}]},{"text":").","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"e4a7ff9b-e0ab-489f-acf2-d96dca06fdfc"}}]},{"type":"hardBreak"}]},{"type":"paragraph","content":[{"type":"hardBreak"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Step 2: Generate the atmosphere initial condition ","type":"text"}]},{"type":"paragraph","content":[{"text":"There are two general approaches to generating a new atmospheric initial condition file: remapping an existing initial condition file or remapping reanalysis data. Both of these situations can be addressed with the new HICCUP tool. ","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Generating atmosphere initial condition data using HICCUP","type":"text"}]},{"type":"paragraph","content":[{"text":"The procedure to generate new initial conditions (","type":"text"},{"text":"outlined here","type":"text","marks":[{"type":"link","attrs":{"href":"https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/1002373272"}}]},{"text":") has been built into into the HICCUP tool (see ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://github.com/E3SM-Project/HICCUP"}},{"text":" ). HICCUP is a set of flexible and robust python routines to automate and streamline the task of generating a new atmospheric initial condition for E3SM. Most of these routines are essentially wrappers to NCO commands, but there are also routines that directly modify the data using xarray. HICCUP is setup to use ERA5 reanlaysis data or regrid an existing atmospheric initial condition file (either horizontal, vertical, or both). HICCUP includes a surface adjustment routine that follows the original adjustment routine published by ECMWF, which was also reproduced in similar tools by Jerry Olson and Wuyin Lin. HICCUP was built with code readability and documentation as a top priority and includes some unit tests. HICCUP has also successfully been used to generate initial conditions for ne1024 grids using large memory analysis nodes available at HPC centers. Development of HICCUP is ongoing and new features are planned, so any feedback or feature requests are welcome. ","type":"text"}]},{"type":"paragraph","content":[{"text":"The spin-up procedure described below is often not needed when using an initial condition derived from ERA5 reanalysis. However, for certain situations a series of spin-up simulations is required for stability. An example of this is when an RCE initial condition needs to be produced from an aqua planet initial condition. ","type":"text"}]},{"type":"heading","attrs":{"level":4},"content":[{"text":"Spinning up the atmosphere","type":"text"}]},{"type":"paragraph","content":[{"text":"The following procedure is copied from the recommendations in Mark and Colin's Google Doc ","type":"text"},{"text":"https://docs.google.com/document/d/1ymlTgKz2SIvveRS72roKvNHN6a79B4TLOGrypPjRvg0/edit","type":"text","marks":[{"type":"link","attrs":{"href":"https://docs.google.com/document/d/1ymlTgKz2SIvveRS72roKvNHN6a79B4TLOGrypPjRvg0/edit"}}]},{"text":" on running new RRM configurations (TODO: clean this up and update):","type":"text"}]},{"type":"paragraph","content":[{"text":"Determining stable timestep values is also complicated by spinup. If the initial condition file is very much out of balance (say it came from real planet or a very different aqua planet simulation), then you may require very small timesteps and larger hyperviscosity coefficients in order to get past an initial adjustment period. Only perform the tuning of se_nsplit and hypervis_subcycle with a spun up initial condition file. ","type":"text"}]},{"type":"paragraph","content":[{"text":"With a high quality grid, usually one can run with the uniform se_nsplit value and a slight increase in hypervis_subcycle. Standalone HOMME or EAM simulations using the new grid will compute this number and output “Max Dinv-based element distortion” in the log file. The equal-angle cubed-sphere grid has a value of 1.7. A high quality regionally refined grid will have a value less than 4. ","type":"text"}]},{"type":"paragraph","content":[{"text":"Recommend procedure:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Run 10 days with very large se_nsplit and hypervis_subcycle options and create a new IC file (see INITHIST in CAM). For aqua planet, this step is often not necessary, but for","type":"text"},{"text":" simulations with topography","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"619bb992-6300-44f1-9502-13a0bdb8c216"}}]},{"text":" this step is critical. One may also need to reduce dtime and increase the viscosity coefficients. In an extreme case, something like this might be needed:","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph"},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"run 5-10 days with dtime 3x smaller than the default (and viscosity coefficients 2x larger)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"restarting from step a, run 5-10 days with dtime 3x smaller and default viscosity coefficients","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"restarting from step b, run 5-10 days with dtime 2x smaller and default viscosity coefficients","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"restarting from step c, run 5-10 days with default dtime and se_nsplit 2x larger ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"restarting from step d, run with all default parameters","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Use this new IC file for all future runs below","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"First we determine a stable value of se_nsplit. To do this, we first ensure the viscosity timestep is not causing problems (this is especially true if nu_div > nu), so start with a large value of hypervis_subcycle, say hypervis_subcycle=20","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Find the smallest value of se_nsplit for which the code is stable using 1 month runs. Start with se_nsplit that is used by the corresponding high resolution uniform grid (assuming they all have the same physics timestep, dtime)","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"One a stable value of se_nsplit has been found, decrease hypervis_subcycle until the smallest stable value is found. Note that these are not independent. You have to find the stable value of se_nsplit before finding the stable value of hypervis_subcycle.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Final step: The procedure outlined above can find timesteps that are borderline unstable, but don’t blow up do to various dissipation mechanisms in CAM. Hence it is a good idea to run 3 months, and look at the monthly mean OMEGA500 from the 3rd month. This field will be noisy, but there should not be any obvious grid artifacts. Weak instabilities can be masked by the large transients in flow snapshots, so it best to look at time averages. ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Note that for simulations with topography, we often increase nu_div. This can trigger a restrictive CFL condition which requires reducing hypervis_subcycle. ","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"During this tuning process, it is useful to compare the smallest ‘dx’ from the atmosphere log file to the smallest ‘dx’ from the global uniform high resolution run. Use the ‘dx’ based on the singular values of Dinv, not the ‘dx’ based on element area. If the ‘dx’ for newmesh.g is 20% smaller than the value from the global uniform grid, it suggests the advective timesteps might need to be 20% smaller, and the viscous timesteps might need to be 44% smaller (they go like dx^2). The code prints out CFL estimates that are rough approximation that can be used to check if you are in the correct ballpark.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"7. Generate land surface data (fsurdat)","type":"text"}]},{"type":"paragraph","content":[{"text":"Requirements:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"land grid descriptor file in SCRIP format ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"ESMF_RegridWeightGen","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"geographic distribution for each land surface type along with grid descriptor files for each of those surface types","type":"text","marks":[{"type":"textColor","attrs":{"color":"#091e42"}}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"mkmapdata.sh","type":"text","marks":[{"type":"strong"}]},{"text":" (found in ","type":"text"},{"text":"components/elm/tools/mkmapdata/","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/E3SM-Project/E3SM/tree/master/components/elm/tools/mkmapdata"}}]},{"text":")","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"mksurfdata.pl","type":"text","marks":[{"type":"strong"}]},{"text":" (found in ","type":"text"},{"text":"components/clm/tools/clm4_5/mksurfdata_map/","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/E3SM-Project/E3SM/tree/master/components/clm/tools/clm4_5/mksurfdata_map"}}]},{"text":")","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Overview:","type":"text"}]},{"type":"paragraph","marks":[{"type":"indentation","attrs":{"level":1}}],"content":[{"text":"A large number of input files need to be interpolated for the land model. This (rather awkward) workflow uses a chain of tools that downloads the inputdata and grid descriptor files for each input dataset, generates mapping files from each input data grid to our target grid, and then applies the mapping weight files to do the interpolation.","type":"text"}]},{"type":"paragraph","content":[{"text":"Notes:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"the steps below are for ","type":"text"},{"text":"maint1-0 code base","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/E3SM-Project/E3SM/tree/maint-1.0"}}]},{"text":". Post-v1 release changes (to add phosphorus) broke existing land initial condition files (finidat) and may require changes to this methodology. ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"the focus here is on creating an fsurdat file in cases where land use land cover change (LULCC) does NOT change. Additional steps will be needed to create a transient LULCC file.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"questions for the land team are in red","type":"text","marks":[{"type":"textColor","attrs":{"color":"#ff5630"}}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"NOTE: The land surface generation tool is currently BROKEN in E3SM master, following addition of FAN and fertilizer mods (PR: ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#ff5630"}}]},{"text":"5981","type":"text","marks":[{"type":"textColor","attrs":{"color":"#ff5630"}},{"type":"link","attrs":{"href":"https://github.com/E3SM-Project/E3SM/pull/5981"}}]},{"text":"). A PR was issued to fix this by removing the unsupported FAN dataset from surfdata files, but was subsequently closed (PR: ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#ff5630"}}]},{"text":"6237","type":"text","marks":[{"type":"textColor","attrs":{"color":"#ff5630"}},{"type":"link","attrs":{"href":"https://github.com/E3SM-Project/E3SM/pull/6237"}}]},{"text":"). For the time being, land surface data can be generated by checking out git hash ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#ff5630"}}]},{"text":"cd22d14bb8","type":"text","marks":[{"type":"code"}]},{"text":" (master immediately before 5981 was merged).","type":"text","marks":[{"type":"textColor","attrs":{"color":"#ff5630"}}]}]}]}]},{"type":"paragraph","content":[{"text":"Steps:","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Create mapping files for ","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"2d3bfb2d-d34b-449f-b2e9-34bc91c96d27"}}]},{"text":"each","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"2d3bfb2d-d34b-449f-b2e9-34bc91c96d27"}},{"type":"em"}]},{"text":" land surface type if needed. An (older and deprecated) example of doing this can be found ","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"2d3bfb2d-d34b-449f-b2e9-34bc91c96d27"}}]},{"text":"here","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"1723d89a-957b-4e3f-9f90-760877667de5"}},{"type":"link","attrs":{"href":"https://acme-climate.atlassian.net/wiki/spaces/LND/pages/17498263/CLM4.5+dataset+for+year+1850+at+ne120np4+resolution"}}]},{"text":". Updated instructions follow:","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"1723d89a-957b-4e3f-9f90-760877667de5"}}]},{"type":"hardBreak"},{"type":"hardBreak"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Obtain or generate a target grid file in SCRIP format. For these example, we will use a ne1024pg2 grid file, which we will need to create (note that most np4 grid files can be found within the inputdata repository, for example, the ne1024np4 grid file is at ","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"1723d89a-957b-4e3f-9f90-760877667de5"}}]},{"text":"https://web.lcrc.anl.gov/public/e3sm/mapping/grids/ne1024np4_scrip_c20191023.nc","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"1723d89a-957b-4e3f-9f90-760877667de5"}},{"type":"link","attrs":{"href":"https://web.lcrc.anl.gov/public/e3sm/mapping/grids/ne1024np4_scrip_c20190125.nc"}}]},{"text":"). To generate the pg2 SCRIP file: ","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"1723d89a-957b-4e3f-9f90-760877667de5"}}]}]},{"type":"codeBlock","content":[{"text":"${tempest_root}/bin/GenerateCSMesh --alt --res 1024 --file ne1024.g\n${tempest_root}/bin/GenerateVolumetricMesh --in ne1024.g --out ne1024pg2.g --np 2 --uniform\n${tempest_root}/bin/ConvertMeshToSCRIP --in ne1024pg2.g --out ne1024pg2_scrip.nc","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Get list of input grid files for each land surface input data file. This is done by running the components/elm/tools/mkmapdata/mkmapdata.sh script in debug mode to output a list of needed files (along with the commands that will be used to generate each map file; also make sure GRIDFILE is set to the SCRIP file from the above step): ","type":"text"}]},{"type":"codeBlock","attrs":{"language":"shell"},"content":[{"text":"cd ${e3sm_root}/components/elm/tools/mkmapdata\n./mkmapdata.sh --gridfile ${GRIDFILE} --inputdata-path ${INPUTDATA_ROOT} --res ne1024pg2 --gridtype global --output-filetype 64bit_offset --debug -v --list","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Download needed input grid files. The above command will output a list of needed files to ","type":"text"},{"text":"clm.input_data_list","type":"text","marks":[{"type":"code"}]},{"text":". We need to download all of these before calling the script without the debug flag to actually perform the mapping. This is possible using ","type":"text"},{"text":"check_input_data","type":"text","marks":[{"type":"code"}]},{"text":" in CIME, but needs to be done from a dummy case directory. So, one can create a dummy case, ","type":"text"},{"text":"cd","type":"text","marks":[{"type":"code"}]},{"text":" to that case, and then call ","type":"text"},{"text":"./check_input_data --data-list-dir --download","type":"text","marks":[{"type":"code"}]},{"text":". However, this failed to connect to the CESM SVN server for me. So instead, I used the following one-off script: ","type":"text"}]},{"type":"codeBlock","content":[{"text":"#!/bin/bash\ne3sm_inputdata_repository=\"https://web.lcrc.anl.gov/public/e3sm\"\ncesm_inputdata_repository=\"https://svn-ccsm-inputdata.cgd.ucar.edu/trunk\"\ninputdata_list=clm.input_data_list\ncat $inputdata_list | while read line; do\n localpath=`echo ${line} | sed 's:.* = \$.*\$:\\1:'`\n url1=${e3sm_inputdata_repository}/`echo ${line} | sed 's:.*\$inputdata/lnd/.*\$:\\1:'`\n url2=${cesm_inputdata_repository}/`echo ${line} | sed 's:.*\$inputdata/lnd/.*\$:\\1:'`\n if [ ! -f ${localpath} ]; then\n echo \"${url1} -> ${localpath}\"\n mkdir -p `dirname ${localpath}`\n cd `dirname ${localpath}`\n # Try to download using first URL, if that fails then use the second\n wget ${url1} || wget ${url2}\n else\n echo \"${localpath} exists, skipping.\"\n fi\ndone","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Create mapping files. Should just be able to run the above ","type":"text"},{"text":"mkmapdata.sh","type":"text","marks":[{"type":"code"}]},{"text":" command without the ","type":"text"},{"text":"–debug --list","type":"text","marks":[{"type":"code"}]},{"text":" flags. We need to append the ","type":"text"},{"text":"--outputfile-type 64bit_offset","type":"text","marks":[{"type":"code"}]},{"text":" flag for our large files (no reason not to do this by default anyways). ","type":"text"},{"text":"NOTE - This step requires NCL, which is no longer part of the E3SM unified environement. If the machine you are using does not have an NCL module, creating a custom environement that includes NCL is an easy work around. Fixing this issue to avoid the NCL dependency will require rewriting the rmdups.ncl and mkunitymap.ncl script in another language (python+xarray would make sense). We will also need to write a version of the gc_qarea() function, unless the geocat project writes a port that we can use (see ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#ff5630"}},{"type":"strong"}]},{"text":"geocat issue #31","type":"text","marks":[{"type":"textColor","attrs":{"color":"#ff5630"}},{"type":"strong"},{"type":"link","attrs":{"href":"https://github.com/NCAR/geocat-comp/issues/31"}}]},{"text":").","type":"text","marks":[{"type":"textColor","attrs":{"color":"#ff5630"}},{"type":"strong"}]}]},{"type":"codeBlock","content":[{"text":"./mkmapdata.sh --gridfile ${GRIDFILE} --inputdata-path ${INPUTDATA_ROOT} --res ne1024pg2 --gridtype global --output-filetype 64bit_offset -v","type":"text"}]}]}]}]}]},{"type":"paragraph","marks":[{"type":"indentation","attrs":{"level":2}}],"content":[{"text":"Notes for Perlmutter (2024) - If you come across an error using ESMF that includes “","type":"text"},{"text":"This functionality requires ESMF to be built with the PIO library enabled","type":"text","marks":[{"type":"code"}]},{"text":"“ in the log file, then this means you are using a version of ESMF that is not compiled with MPI support. If using a conda env then you can check the type of build using ","type":"text"},{"text":"conda list | grep esmf","type":"text","marks":[{"type":"code"}]},{"text":" and look for “mpi” or “nompi” in the build tag. The ESMF library can also be problematic when creating the “HYDRO1K“ map (make sure to use the SCRIP version of that grid!). Recent attempts to create this map resulted in segmentation faults, which seem to be a problem only in recent versions. In particular, version 8.6 was found to be problematic, but rolling back to version 8.0.1 was successful. ","type":"text"}]},{"type":"orderedList","attrs":{"order":2},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Compile surface dataset source code ","type":"text"},{"type":"hardBreak"},{"text":"NOTE: previously, editing the makefile ","type":"text"},{"text":"${e3sm_root}/components/elm/tools/mksurfdata_map/src/Makefile.common","type":"text","marks":[{"type":"code"}]},{"text":" was required on most machines, but this was fixed by ","type":"text"},{"text":"https://github.com/E3SM-Project/E3SM/pull/2757","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/E3SM-Project/E3SM/pull/2757"}}]},{"text":"):","type":"text"}]},{"type":"codeBlock","content":[{"text":"# Setup environment (should work on any E3SM-supported machine)\neval $(${e3sm_root}/cime/CIME/Tools/get_case_env)\n${e3sm_root}/cime/CIME/scripts/configure --macros-format Makefile --mpilib mpi-serial\nsource .env_mach_specific.sh\n\n# Build mksurfdata_map\ncd ${e3sm_root}/components/elm/tools/mksurfdata_map/src\nINC_NETCDF=\"`nf-config --includedir`\" \\\n LIB_NETCDF=\"`nc-config --libdir`\" USER_FC=\"`nc-config --fc`\" \\\n USER_LDFLAGS=\"`nf-config --flibs`\" make\n","type":"text"}]},{"type":"paragraph","content":[{"text":"Note for Perlmutter (Mar 2024 - Walter Hannah) - the build worked using the command below:","type":"text"}]},{"type":"codeBlock","content":[{"text":"cd ${e3sm_root}/components/elm/tools/mksurfdata_map/src\n${e3sm_root}/cime/CIME/scripts/configure && source .env_mach_specific.sh\nmake clean\nINC_NETCDF=\"`nf-config --includedir`\" \\\nLIB_NETCDF=\"`nc-config --libdir`\" \\\nUSER_FC=\"`nc-config --fc`\" \\\nUSER_LDFLAGS=\"-L`nc-config --libdir` -lnetcdf -lnetcdff\" \\\nmake","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Run the ","type":"text"},{"text":"mksurfdata.pl","type":"text","marks":[{"type":"link","attrs":{"href":"http://mksurfdata.pl"}}]},{"text":" script in \"debug\" mode to generate the namelist (use year 2010 on ne120np4 grids as an example). ","type":"text"}]},{"type":"codeBlock","content":[{"text":"# For supported resolutions\n#(use year 2010 on ne120np4 grids as an example)\ncd $e3sm_dir/components/elm/tools/mksurfdata_map\n./mksurfdata.pl -res ne120np4 -y 2010 -d -dinlc /global/cfs/cdirs/e3sm/inputdata -usr_mapdir /global/cfs/cdirs/e3sm/inputdata/lnd/clm2/mappingdata/maps/ne120np4\n\n# For unsupported, user-specified resolutions\n# (use year 2010 on ne50np4 grid as an example)\n# (Assuming the mapping files created in step 1 has a time stamp of '190409' in the filenames and the location of mapping files are '/whatever/directory/you/put/mapping/files')\n./mksurfdata.pl -res usrspec -usr_gname ne50np4 -usr_gdate 190409 -y 2010 -d -dinlc /global/cfs/cdirs/e3sm/inputdata -usr_mapdir /whatever/directory/you/put/mapping/files","type":"text"}]},{"type":"paragraph","content":[{"text":"(However, ./","type":"text"},{"text":"mksurfdata.pl","type":"text","marks":[{"type":"link","attrs":{"href":"http://mksurfdata.pl"}}]},{"text":" -h shows -y is by default 2010. When running without \"-y\" option, standard output says sim_year 2000. I suspect the ","type":"text"},{"text":"mksurfdata.pl","type":"text","marks":[{"type":"link","attrs":{"href":"http://mksurfdata.pl"}}]},{"text":" help information is wrong. ","type":"text"},{"text":"To be confirmed","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"68bf0241-59ec-4258-93b8-0620d42ed05e"}},{"type":"textColor","attrs":{"color":"#ff0000"}}]},{"text":".","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"68bf0241-59ec-4258-93b8-0620d42ed05e"}}]},{"text":")","type":"text"},{"type":"hardBreak"},{"type":"hardBreak"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Modify namelist file ","type":"text"},{"type":"hardBreak"},{"text":"(","type":"text"},{"text":"Should the correct namelist settings be automatically picked up if the default land build name list settings are modified accordingly?","type":"text","marks":[{"type":"textColor","attrs":{"color":"#ff0000"}}]},{"text":")","type":"text"}]},{"type":"paragraph","content":[{"text":"Time-evolving Land use land cover change (LULCC) data should not be used for fixed-time compsets, but the LULCC information for that particular year should be used (","type":"text"},{"text":"right?","type":"text","marks":[{"type":"textColor","attrs":{"color":"#ff0000"}}]},{"text":")","type":"text"},{"type":"hardBreak"},{"text":"Manually change to mksrf_fvegtyp = '/global/cfs/cdirs/e3sm/inputdata/lnd/clm2/rawdata/AA_mksrf_landuse_rc_185","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"0c648aa7-ca34-4627-9e03-8a3354d2c0f0"}}]},{"text":"0-2015_06062017_LUH2/","type":"text"},{"text":"AA_mksrf_landuse_rc_2010_06062017.nc","type":"text","marks":[{"type":"link","attrs":{"href":"http://AA_mksrf_landuse_rc_2010_06062017.nc"}}]},{"text":"' for the F2010 ne120 compset.","type":"text"},{"type":"hardBreak"},{"type":"hardBreak"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Create the land surface data by interactive or batch job","type":"text"}]},{"type":"codeBlock","content":[{"text":"rm -f surfdata_ne120np4_simyr2010.bash\ncat <> surfdata_ne120np4_simyr2010.bash\n#!/bin/bash\n\n#SBATCH --job-name=mksurfdata2010\n#SBATCH --account=acme\n#SBATCH --nodes=1\n#SBATCH --output=mksurfdata.o%j\n#SBATCH --exclusive\n#SBATCH --time=00:30:00\n#SBATCH --qos=debug\n\n# Load modules\nmodule load nco\nmodule load ncl\nmodule load cray-netcdf\nmodule load cray-hdf5\n\n# mksurfdata_map is dynamically linked\nexport LIB_NETCDF=$NETCDF_DIR/lib\nexport INC_NETCDF=$NETCDF_DIR/include\nexport USER_FC=ifort\nexport USER_CC=icc\nexport USER_LDFLAGS=\"-L$NETCDF_DIR/lib -lnetcdf -lnetcdff -lnetcdf_intel\"\nexport USER_LDFLAGS=$USER_LDFLAGS\" -L$HDF5_DIR/lib -lhdf5 -lhdf5_fortran -lhdf5_cpp -lhdf5_fortran_intel -lhdf5_hl_intel -lhdf5hl_fortran_intel\"\n\ncd /global/homes/t/tang30/ACME_code/MkLandSurf/components/clm/tools/clm4_5/mksurfdata_map\n\nCDATE=c`date +%y%m%d` # current date\n\n./mksurfdata_map < namelist\nEOF\n\nsbatch surfdata_ne120np4_simyr2010.bash","type":"text"}]},{"type":"paragraph","content":[{"text":"The land surface data in NetCDF format will be created at current directory. (","type":"text"},{"text":"How to verify the file is correct?","type":"text","marks":[{"type":"textColor","attrs":{"color":"#ff0000"}}]},{"text":")","type":"text"}]}]}]},{"type":"paragraph","content":[{"type":"hardBreak"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"8. Generate a new land initial condition (finidat)","type":"text"}]},{"type":"paragraph","content":[{"text":"Three options:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"cold start: finidat=\"\", no file necessary. Lets us get up and running, but not suitable for climate science applications","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Interpolate a spunup state from a previous simulation. This is reasonable for many applications, but not suitable for official published E3SM simulations.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"spin-up a new initial condition following best practices from land model developers. ","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"From ","type":"text"},{"type":"mention","attrs":{"id":"557058:8e83e32a-4a28-4548-81cd-ae9d1d94964d","text":"@Peter Thornton"}},{"text":" via email. We may end up changing what we say, but this is a start.","type":"text"}]},{"type":"blockquote","content":[{"type":"paragraph","content":[{"text":"what I would recommend in general, is to start the spin-up process with a land cold-start condition, using reanalysis data atmosphere and having all the other land settings just the way you want them for the eventual coupled simulation (resolution, domain file, land physics and BGC settings). Then run for enough years to get an approximate steady state (the number depends on what land options you are using – no BGC means a lot fewer years). Then use the resulting restart file as the initial condition for a coupled run that has at least the same atmosphere settings you will eventually use for your production run. Save high-frequency output for multiple years (preferably 10 or more, but a high res run might not have that luxury). Use that atm output to drive a second offline land spin-up, to equilibrate the land to the expected initial climate from the atmosphere. Then use the resulting land restart as finidat for the start of your fully-coupled spin-up simulation, and let it run for a while to assess drifts in all coupled components. Only after you are satisfied that everything is stable are you in a safe state to begin a science experiment production run. Off-the-shelf finidat file is not likely to save you much time in this process, because it will not be spun up to your experimental conditions and tolerances.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"9. Create a new atmospheric dry deposition file","type":"text"}]},{"type":"paragraph","content":[{"text":"From the README for ","type":"text"},{"text":"mkatmsrffile","type":"text","marks":[{"type":"code"}]},{"text":" tool at ","type":"text"},{"text":"components/eam/tools/mkatmsrffile","type":"text","marks":[{"type":"code"}]},{"text":":","type":"text"}]},{"type":"paragraph","marks":[{"type":"indentation","attrs":{"level":1}}],"content":[{"text":"Atmospheric drydeposition at the surface depends on certain surface","type":"text"},{"type":"hardBreak"},{"text":"properties including soil and land use properties. In most cases","type":"text"},{"type":"hardBreak"},{"text":"these calculations can be handled in the land model and passed to he","type":"text"},{"type":"hardBreak"},{"text":"atmosphere through the coupler. This is the default namelist setting","type":"text"},{"type":"hardBreak"},{"text":"drydep_method='xactive_lnd'. However with modal areosols this method","type":"text"},{"type":"hardBreak"},{"text":"is not adequate and we must recalculate these fields in the atmosphere","type":"text"},{"type":"hardBreak"},{"text":"(see subroutine interp_map in mo_drydep.F90). For unstructured grids","type":"text"},{"type":"hardBreak"},{"text":"it was determined to create this offline interpolation tool rather","type":"text"},{"type":"hardBreak"},{"text":"than generalize the subroutine interp_map.","type":"text"}]},{"type":"paragraph","content":[{"text":"Following are the steps for building and running an executable for this tool:","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Change directory to tool root:","type":"text"},{"type":"hardBreak"},{"text":"cd components/eam/tools/mkatmsrffile","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Create a .env_mach_specific.sh by running ","type":"text"},{"type":"hardBreak"},{"text":"../../../../cime/tools/configure --macros-format=Makefile","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Get machine-specific environment settings via ","type":"text"},{"type":"hardBreak"},{"text":"source .env_mach_specific.sh","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Make sure ","type":"text"},{"text":"NETCDF_ROOT","type":"text","marks":[{"type":"code"}]},{"text":" and ","type":"text"},{"text":"FC","type":"text","marks":[{"type":"code"}]},{"text":" environment variables are set right for your system, and build the executable:","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"On Cori: ","type":"text"},{"text":"env NETCDF_ROOT=$NETCDF_DIR FC=ifort make","type":"text","marks":[{"type":"code"}]}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Edit \"nml_atmsrf\" to update the input file paths","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Run the newly built executable","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"}]},{"type":"codeBlock","content":[{"text":"./mkatmsrffile","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"}]}]}]},{"type":"paragraph","content":[{"text":" This will produce a drydep file. Following input files were used for generating a new dry deposition file:","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"}]},{"type":"paragraph","content":[{"text":"srfFileName: /project/projectdirs/e3sm/mapping/grids/","type":"text"},{"text":"1x1d.nc","type":"text","marks":[{"type":"link","attrs":{"href":"http://1x1d.nc"}}]}]},{"type":"paragraph","content":[{"text":"atmFileName: /project/projectdirs/e3sm/mapping/grids/","type":"text"},{"text":"ne30np4_pentagons.091226.nc","type":"text","marks":[{"type":"link","attrs":{"href":"http://ne30np4_pentagons.091226.nc"}}]}]},{"type":"paragraph","content":[{"text":"landFileName: /project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/dvel/","type":"text"},{"text":"regrid_vegetation.nc","type":"text","marks":[{"type":"link","attrs":{"href":"http://regrid_vegetation.nc"}}]}]},{"type":"paragraph","content":[{"text":"soilwFileName: /project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mozart/dvel/","type":"text"},{"text":"clim_soilw.nc","type":"text","marks":[{"type":"link","attrs":{"href":"http://clim_soilw.nc"}}]}]},{"type":"paragraph","content":[{"text":"srf2atmFmapname:/project/projectdirs/e3sm/bhillma/grids/ne30np4/atmsrffile/","type":"text"},{"text":"map_1x1_to_ne30np4_aave.nc","type":"text","marks":[{"type":"link","attrs":{"href":"http://map_1x1_to_ne30np4_aave.nc"}}]}]},{"type":"paragraph","content":[{"text":"Note that if using Tempest Remap to provide mapping files, the above mapping file should be replaced with something that looks like ","type":"text"},{"text":"map_1x1_to_ne30np4_","type":"text","marks":[{"type":"code"}]},{"text":"mono","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"2f52e5e0-1b99-446d-aba0-b0a4e7b681ff"}},{"type":"code"}]},{"text":".nc","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"Output file produced using the above procedure was compared against an existing file (/project/projectdirs/e3sm/inputdata/atm/cam/chem/trop_mam/","type":"text"},{"text":"atmsrf_ne30np4_110920.nc","type":"text","marks":[{"type":"link","attrs":{"href":"http://atmsrf_ne30np4_110920.nc"}}]},{"text":") using a script from ","type":"text"},{"type":"mention","attrs":{"id":"557058:4dbc3900-5a65-4e7c-85b0-5da72308a6fb","text":"@Peter Caldwell"}},{"text":". Following figures show the comparison:","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"align-start","width":25.0},"content":[{"type":"media","attrs":{"width":640,"id":"aee9a4a6-a913-42a1-9f7c-df990729e096","collection":"contentId-872579110","type":"file","height":480}}]},{"type":"mediaSingle","attrs":{"layout":"align-start","width":25.0},"content":[{"type":"media","attrs":{"width":640,"id":"f9fff371-098e-41ca-a8e0-d1c879d93674","collection":"contentId-872579110","type":"file","height":480}}]},{"type":"paragraph","content":[{"type":"hardBreak"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"10. Create a new compset and/or new supported grid by modifying CIME's xml files","type":"text"}]},{"type":"paragraph","content":[{"text":"???","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"11. Implement tests for the new grid","type":"text"}]},{"type":"paragraph","content":[{"text":"(TODO: develop this section. One example might be to include a case-building test as done in ${e3sm_root}/cime/config/e3sm/tests.py)","type":"text"}]},{"type":"paragraph"},{"type":"heading","attrs":{"level":2},"content":[{"text":"12. Adding a new ocean and sea-ice mesh","type":"text"}]},{"type":"paragraph","content":[{"text":"If you wish to add a new ocean and sea-ice mesh in addition to (or instead of) a new atmosphere grid, you will need to use the ","type":"text"},{"text":"compass","type":"text","marks":[{"type":"code"}]},{"text":" tool to generate the mesh and dynamically adjusted initial condition. This procedure is detailed in a separate tutorial:","type":"text"},{"type":"hardBreak"},{"type":"inlineCard","attrs":{"url":"https://mpas-dev.github.io/compass/latest/tutorials/dev_add_rrm.html"}},{"text":" ","type":"text"}]},{"type":"paragraph","content":[{"text":"The new ocean/sea ice mesh should be generated before generating mapping and domain files (steps 3. and 4. above).","type":"text"},{"type":"hardBreak"}]},{"type":"rule"},{"type":"heading","attrs":{"level":1},"content":[{"text":"Summary","type":"text"}]},{"type":"paragraph","content":[{"text":"After reading through the info above on ","type":"text"},{"type":"date","attrs":{"timestamp":"1546905600000"}},{"text":" , I (","type":"text"},{"type":"mention","attrs":{"id":"557058:4dbc3900-5a65-4e7c-85b0-5da72308a6fb","text":"@Peter Caldwell"}},{"text":") created lists of stuff we should create tests for, merge to master, and revise to avoid dual-grid dependency. ","type":"text"},{"type":"mention","attrs":{"id":"557058:8321d702-0b82-4350-8663-c999fc45ed34","text":"@Ben Hillman"}},{"text":" - am I missing anything?","type":"text"}]},{"type":"paragraph","content":[{"text":"Tools we should create tests for:","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"TempestRemap","type":"text","marks":[{"type":"strong"},{"type":"link","attrs":{"href":"https://github.com/ClimateGlobalChange/tempestremap"}}]},{"text":" for generating uniform grids ","type":"text"},{"type":"hardBreak"},{"text":"(in Paul’s external git repo - may have its own tests?)","type":"text","marks":[{"type":"em"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"SQuadGen","type":"text","marks":[{"type":"strong"}]},{"text":" for generating RRM grids ","type":"text"},{"type":"hardBreak"},{"text":"(in Paul’s external repo - may have its own tests?)","type":"text","marks":[{"type":"em"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Generate topography via ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/2720202817"}},{"text":" )","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"needs utilities: components/eam/tools/topo_tool/cube_to_target and comopnents/homme/test/tool","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"run ","type":"text"},{"text":"ncremap","type":"text","marks":[{"type":"strong"}]},{"text":" (an NCO command) to generate mapping files","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"cime/tools/mapping/gen_domain_files","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"mksurfdata.pl","type":"text","marks":[{"type":"link","attrs":{"href":"http://mksurfdata.pl"}}]},{"text":" to generate the namelist needed to make fsurdat file","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"use ","type":"text"},{"text":"mksurfdata_map","type":"text","marks":[{"type":"strong"}]},{"text":" for fsurdat","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"use the ","type":"text"},{"text":"interpic_new","type":"text","marks":[{"type":"annotation","attrs":{"annotationType":"inlineComment","id":"1c3e0484-a527-4d19-af15-d69fc2c66565"}},{"type":"strong"}]},{"text":" tool to regrid atmos state to new grid for initial condition","type":"text"}]}]}]},{"type":"paragraph","content":[{"type":"hardBreak"}]},{"type":"paragraph","content":[{"text":"Stuff on branches that we need to get on master:","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"branch brhillman/add-grid-scripts for the matlab script used to create the dual grid.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"PR #2633","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/E3SM-Project/E3SM/pull/2633"}}]},{"text":" to generate domain files without needing the dual grid?","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"PR #2706","type":"text","marks":[{"type":"link","attrs":{"href":"https://github.com/E3SM-Project/E3SM/pull/2706"}}]},{"text":" to add command line interface to topography tool to not have to edit source code by hand and recompile to compute subgrid surface roughness","type":"text"}]}]}]},{"type":"paragraph","content":[{"type":"hardBreak"}]},{"type":"paragraph","content":[{"text":"Stuff requiring ESMF and/or the dual grid:","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"generate mapping files needs ESM_GegridWeightGen?","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"generate topography file","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Tools that could use some clean-up:","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"smoothtopo.job","type":"text","marks":[{"type":"code"}]},{"text":" script used to run HOMME to apply dycore-specific smoothing to interpolated topography. It would be nice for this to be able to run via command line arguments rather than having to edit the script (which should make this easier to include in an automated workflow), and we should remove dependence on NCL since this is not guaranteed to be available.","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Replaced with “homme_tool”, 2020/5. see (see ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/2720202817"}},{"text":" )","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"makegrid.job","type":"text","marks":[{"type":"code"}]},{"text":" script used to run HOMME+NCL to produce the non-optimized dualgrid and latlon descriptions of the SE grid. Again, it would be nice for this to be able to run via command line arguments rather than having to edit the script (which should make this easier to include in an automated workflow), and we should remove dependence on NCL since this is not guaranteed to be available.","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"TR and PG2 grids make this obsolete - we now longer need the “dualgrid”. ","type":"text"}]}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Land surface data scripts (","type":"text"},{"text":"TODO: add specifics about what needs to change here","type":"text","marks":[{"type":"textColor","attrs":{"color":"#ff0000"}}]},{"text":")","type":"text"}]}]}]},{"type":"paragraph","content":[{"type":"hardBreak"}]},{"type":"paragraph","content":[{"type":"hardBreak"}]}],"version":1}