DRAFT: E3SM-Unified 1.11.0 release notes

Dear E3SM Team,

A new release of E3SM-Unified is available: 1.11.0. E3SM-Unified is a combination conda and spack environment that includes a large amount of analysis and pre- and post-processing software for E3SM developers and users.

Notable Updates:

ChemDyg, E3SM-Diags, e3sm_to_cmip, LIVVKit, MPAS-Analysis, NCO, xcdat, zppy, and zstash

New Packages:

CDO, cime_gen_domain, mosaic, squadgen, zppy-interfaces

NOTE: This will be the last release to support the CDAT packages: cdms2, cdtime, cdutil, genutil, etc. If you use these tools, please find alternatives (notably xarray/xcdat) because they will no longer be supported or maintained.

Activation

As in previous versions, you can access the environment as usual by sourcing an activation script:

Andes:

source /ccs/proj/cli115/software/e3sm-unified/load_latest_e3sm_unified_andes.sh

Anvil:

source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_anvil.sh

Chicoma:

source /usr/projects/e3sm/e3sm-unified/load_latest_e3sm_unified_chicoma-cpu.sh

Chrysalis:

source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh

Compy:

source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh

Dane:

Frontier:

source /ccs/proj/cli115/software/e3sm-unified/load_latest_e3sm_unified_frontier.sh

Perlmutter (login or CPU nodes):

source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh

ALCF Polaris:

source /lus/grand/projects/E3SMinput/soft/e3sm-unified/load_latest_e3sm_unified_polaris.sh

Ruby:

Details

The new version has been deployed on all supported machines: Andes, Anvil, Chicoma, Chrysalis, Compy, Dane, Frontier, Perlmutter, ALCF Polaris (not to be confused with the E3SM Polaris software) and Ruby.

Note: We encourage users at OLCF to use Andes, rather than Frontier, for processing and analysis.

On 8 machines (Anvil, Chicoma, Chrysalis, Compy, Dane, Frontier, Perlmutter and Ruby) there are 6 packages of interest -- ESMF, ILAMB, MOAB, NCO, TempestExtremes and TempestRemap -- that have been built with Spack using system compilers and MPI libraries. When you load E3SM-Unified on a compute node, you will have access to these versions, which can be run in parallel and which will typically run more efficiently than their counterparts in conda packages.

The tools needed to build E3SM documentation locally with mkdoc are also included to E3SM-Unified.

Important Updates in 1.11.0

ChemDyg 1.1.0:

Updates to be compatible with zppy (v3.0.0)
Some minor fixes

E3SM-Diags 3.0.0 :

v3.0.0 marks a major milestone after nearly two years of work by the core development team. This release introduces a completely new back-end, replacing CDAT with Xarray and xCDAT. The user-facing API for running E3SM Diagnostics remains backward-compatible between v2 and v3. The modernization improves performance, usability, and maintainability, unit tests coverage and regression testing framework.
User-facing Enhancements
- Add support for most EAMxx variables.
- Replaced SciPy cwt module with PyWavelets cwt for the added wavelet analysis in QBO set.
- The viewer is now at the top level of results directory
- Revamped logger module for additional features and bug fixes.
View the full changelog [TBA] for more information.

e3sm_to_cmip 1.11.3:

Bug Fixes to fix table/freq ambiguity for multi-frequency variables; Remove positive from hfsifrazil; Fix masking of 3D MPAS-Ocean variables.
Additional refactoring of cmor setup for specific variables (areacella, clisccp, orog, sftlf).
Consolidate cmor setup code to simplify implementation.

e3sm-tools 3.0.2:

Updates to cube_to_target, interpinic, mksurfdata_map as part of the E3SM v3.0.2 tag

LIVVKit 3.1.0:

Handles missing performance files more gracefully
Improves file diff section header titles

MPAS-Analysis 1.12.0 and 1.13.0:

Maps of surface mass and heat fluxes
Maps of velocity
Maps of numerous new sea-ice fields
Major bug fix to the horizontal streamfunction

NCO 5.3.2:

Vertical interpolation works on latest ERA5, MERRA2 formats
ncclimo timeseries will output regional statistics (scaled sums or means)
ELM restart file gridification allows selecting output landunit/column type, re-orders snow layers
ncks --chk_map diagnoses grid resolution
ncremap regridder improvements, MPAS-A support

xcdat 0.8.0:

Improve performance of the Regrid2 API.
Significantly improved the performance of temporal group averaging by addressing an Xarray bug discovered during debugging. The bug was subsequently fixed by the Xarray team.
Enhanced support for temporal averaging with custom seasons
Many bug fixes and documentation updates.
View the full changelog for more information.

zppy 3.0.0:

zppy 3.0.0 is a major release; there are non-backwards-compatible ("breaking") changes. It is therefore highly recommended to read the release notes (INSERT LINK HERE), in particular for users of e3sm_to_cmip, tc_analysis, global_time_series, and ilamb functionalities.
The e3sm_to_cmip functionality is now accessed via its own task rather than via the ts task with ts_fmt = cmip.
The global_time_series task is now implemented with a new package, zppy-interfaces, which is meant for Python scripts that extend external packages for easy use by E3SM users, whereas zppy itself is meant for workflow automation (i.e., calling code written outside zppy).
global_time_series has also been extended to work for Land variables and to generate viewers similar to those in e3sm_diags.
Parameter inference and validation have been improved and formalized. Users now have the ability to specify if they don't want zppy to infer parameters (for a file path or for the section name of a task dependency).
CDAT is no longer a dependency for zppy. xCDAT is now user for handling global_time_series and e3sm_diags.
The tc_analysis task underwent a number of improvements to be more robust.

zstash 1.4.4:

Important changes were made to the zstash code base to make it more robust:
- --follow-symlinks will now only create a symlink in the destination directory, not also the source directory.
- --non-blocking functionality (i.e., tars are created while a Globus transfer is ongoing) now occurs if and only if this option is used, as opposed to always occurring.

New in 1.11.0

cime_gen_domain 6.1.59:

The gen_domain tool as part of the CIME v6.1.59 tag, now separate from e3sm-tools.

mosaic 1.1.0:

Enables visualization of unstructured MPAS data on the native mesh with matplotlib, through drop in replacements for pcolormesh / pcolor.
Visualization of fields at all mesh locations (i.e. Cells, Edges, and Vertices) is supported.
Supports map projected figures through cartopy for both planar and spherical meshes.
- Currently only a subset of map projections are supported for spherical meshes, but we are currently working on expanding support to all cartopy projections.

squadgen 1.2.2:

Now separate from e3sm-tools.

zppy-interfaces 0.1.1:

As noted in the zppy section above, this is a new package meant for Python scripts that extend external packages for easy use by E3SM users. It is currently home to the global_time_series plotting functionality. Future additions include the PCMDI metrics plotting.

Next version

Testing of the next version (1.12.0) is planned to begin August with deployment is expected in September, 2025. To request packages and versions to include, make a comment on Next Version.

As always, please email me if there are any questions, or post an issue at https://github.com/E3SM-Project/e3sm-unified/issues.

Cheers,

Xylar and the Infrastructure Team