Data

Tools

Indexes

Applications

Experiments

Open Sustainable Technology

Recent Releases of E3SM Diagnostics Package

E3SM Diagnostics Package - v3.0.0 Modernization with Xarray and xCDAT

What's Changed

v3.0.0 marks a major milestone after nearly two years of work by the core development team. This release introduces a completely new back-end, replacing CDAT with Xarray and xCDAT. Due to the significant scale of code changes, this has been incremented as a major release. The user-facing API for running E3SM Diagnostics remains backward-compatible between v2 and v3.

The modernization improves performance, usability, and maintainability, paving the way for future enhancements to E3SM development. The refactored codebase is now more robust and extensively covered by unit tests, setting a solid foundation for ongoing testing and development.

This release also includes several new enhancements including support for more EAMxx variables, standardization of regridding with xESMF's conservative_normed method, and improving diagnostic run performance by skipping unnecessary regridding of datasets with the same lat-lon grid.

We are excited to share this release with the E3SM community and remain committed to continuously improving the software!

Enhancements

Bug Fixes

Refactor and testing

DevOps

Full Changelog: https://github.com/E3SM-Project/e3sm_diags/compare/v2.12.1...v3.0.0

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by tomvothecoder 2 months ago

E3SM Diagnostics Package - v3.0.0rc4

Refer to the v3.0.0 changelog for more information.

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by tomvothecoder 2 months ago

E3SM Diagnostics Package - v3.0.0rc3

Refer to the v3.0.0 changelog for more information.

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by tomvothecoder 3 months ago

E3SM Diagnostics Package - v3.0.0rc2

Refer to the v3.0.0 changelog for more information.

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by tomvothecoder 4 months ago

E3SM Diagnostics Package - v3.0.0rc1

Refer to the v3.0.0 changelog for more information.

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by tomvothecoder 4 months ago

E3SM Diagnostics Package - v2.12.1 A Fond Farewell to CDAT

As we release version v2.12.1 of E3SM Diags, we would like to take a moment to acknowledge and express our gratitude to the developers and contributors from the CDAT (Climate Data Analysis Tools) community. Over the years, CDAT has been an invaluable tool for our development and analysis processes. This release marks a milestone, as it will be the final version of E3SM Diags that includes a dependency on CDAT.

Looking ahead, E3SM Diags will transition to utilizing xarray/xCDAT tools, which align with our future goals and enhances the performance and usability of our software. We are committed to ensuring a smooth transition for our users and will provide backward compatibility in upcoming releases.

User-facing changes

Other Update

New Contributors

Full Changelog: https://github.com/E3SM-Project/e3sm_diags/compare/v2.12.0...v2.12.1

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by chengzhuzhang 6 months ago

E3SM Diagnostics Package - v2.12.0

New Feature

Updates and bug fixes

Full Changelog: https://github.com/E3SM-Project/e3sm_diags/compare/v2.11.0...v2.12.0

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by tomvothecoder about 1 year ago

E3SM Diagnostics Package - v2.12.0rc1

New Feature

Updates and bug fixes

Full Changelog: https://github.com/E3SM-Project/e3sm_diags/compare/v2.11.0...v2.12.0rc1

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by chengzhuzhang about 1 year ago

E3SM Diagnostics Package - v2.11.0

What's Changed

New Features

Documentation Updates

Bug fixes

Others

Full Changelog: https://github.com/E3SM-Project/e3sm_diags/compare/v2.10.1...v2.11.0

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by tomvothecoder about 1 year ago

E3SM Diagnostics Package - v2.11.0rc1

What's Changed

New Features

Documentation Updates

Bug fixes

Others

Full Changelog: https://github.com/E3SM-Project/e3sm_diags/compare/v2.10.1...v2.11.0rc1

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by chengzhuzhang over 1 year ago

E3SM Diagnostics Package - v2.10.1

What's Changed

Bug Fixes

Refactor

Full Changelog: https://github.com/E3SM-Project/e3sm_diags/compare/v2.10.0...v2.10.1

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by tomvothecoder over 1 year ago

E3SM Diagnostics Package - v2.10.1rc1

What's Changed

Full Changelog: https://github.com/E3SM-Project/e3sm_diags/compare/v2.10.0...v2.10.1rc1

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by tomvothecoder over 1 year ago

E3SM Diagnostics Package - v2.10.0

What's Changed

Features

Bug Fixes

Refactor

Documentation

DevOps

Full Changelog: https://github.com/E3SM-Project/e3sm_diags/compare/v2.9.0...v2.10.0

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by tomvothecoder over 1 year ago

E3SM Diagnostics Package - v2.9.0

New Features

CDAT Migration:

Other Changes:

New Contributors

Full Changelog: https://github.com/E3SM-Project/e3sm_diags/compare/v2.8.0...v2.9.0

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by chengzhuzhang over 1 year ago

E3SM Diagnostics Package - v2.9.0rc5

What's Changed

Full Changelog: https://github.com/E3SM-Project/e3sm_diags/compare/v2.9.0rc4...v2.9.0rc5

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by chengzhuzhang over 1 year ago

E3SM Diagnostics Package - v2.9.0rc4

What's Changed

Full Changelog: https://github.com/E3SM-Project/e3sm_diags/compare/v2.9.0rc3...v2.9.0rc4

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by chengzhuzhang over 1 year ago

E3SM Diagnostics Package - v2.9.0rc3

What's Changed

Full Changelog: https://github.com/E3SM-Project/e3sm_diags/compare/v2.9.0rc2...v2.9.0rc3

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by chengzhuzhang almost 2 years ago

E3SM Diagnostics Package - v2.9.0rc2

What's Changed

Full Changelog: https://github.com/E3SM-Project/e3sm_diags/compare/v2.9.0rc1...v2.9.0rc2

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by chengzhuzhang almost 2 years ago

E3SM Diagnostics Package - v2.9.0rc1

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by tomvothecoder almost 2 years ago

E3SM Diagnostics Package - v2.8.0

New Features

Other Changes

Full Changelog: https://github.com/E3SM-Project/e3sm_diags/compare/v2.7.0...v2.8.0

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by tomvothecoder over 2 years ago

E3SM Diagnostics Package - v2.8.0rc1 Aerosols

New Features

Other Changes

Full Changelog: https://github.com/E3SM-Project/e3sm_diags/compare/v2.7.0...v2.8.0rc1

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by chengzhuzhang over 2 years ago

E3SM Diagnostics Package - v2.7.0

New Features:

  • Added scatter plots for comparing annual mean long-term Absorption Optical Depth (AOD) Aerosol Absorption Optical Depth (AAOD) at 550 nm from AERONET measurements. Scripts and data contributed by @yfenganl
  • Add cmip6 comparison box whisker plot. Scripts contributed by @golaz
  • Add CMIP6 models + E3SM v1 v2, AMIP and HISTORICAL runs as baseline for box whisker plots and Taylor Diagrams
  • Support Model only analysis in lat-lon set
  • Add lat_lon_land annd lat_lon_river drivers and viewers supported in the main viewer.
  • Integrate in ELM land variable pre-processing and MOSART river variables for lat-lon model vs model default run. (contributed by @darincomeau @jenniferholm and @hydrotian )
  • Add a standalone script to generate a summary plot of E3SM diagnostics. by @ambrad
  • Included a Metrics checker to check consistency between two lat-lon run.

Enhanced Features:

  • Add AODDUST to lat_lon plots (model only)
  • Add Q to zonal_mean_2d plots
  • Add TREFMNAV/tasmin TREFMXAV/tasmax to core set
  • Replace ERA Interim with ERA5 in core sets
  • Update to MACv2 for AOD at 550nm diags

Major Fixes:

  • Set regrid method to bilinear for a subset of variables with masks or missing data
  • Fix and Enhancement for Diurnal Cycle of Precipitation
  • Fix extreme values of Albedo from obs values over high latitudes by masking high latitude in lat-lon plot

Other Updates:

  • Update DevOps tooling; Update black to fix click ImportError
  • Add xarray and mache to conda env dependencies and remove dask
  • Update input data path; nco invocation in pre-processing
  • Fix FileNotFoundError for log file

Contributors

  • @chengzhuzhang @tomvothecoder @golaz @forsyth2 @ambrad @yfenganl @darincomeau @jenniferholm and @hydrotian

Full Changelog: https://github.com/E3SM-Project/e3sm_diags/compare/v2.6.1...v2.7.0

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by chengzhuzhang almost 3 years ago

E3SM Diagnostics Package - v2.7.0rc2

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by chengzhuzhang almost 3 years ago

E3SM Diagnostics Package - v2.7.0rc1

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by chengzhuzhang almost 3 years ago

E3SM Diagnostics Package - v2.6.1

What's Changed

Full Changelog: https://github.com/E3SM-Project/e3sm_diags/compare/v2.6.0...v2.6.1

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by tomvothecoder over 3 years ago

E3SM Diagnostics Package - v2.6.1rc1

What's Changed

Full Changelog: https://github.com/E3SM-Project/e3sm_diags/compare/v2.6.0...v2.6.1rc1

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by tomvothecoder over 3 years ago

E3SM Diagnostics Package - v2.6.0

From v2.6.0, Module name acme_diags is renamed to e3sm_diags, users should update their Python run scripts to use import e3sm_diags instead.

Feature

  • Add pressure-latitude set focusing on Stratosphere #535
  • Refine plevs to include stratosphere #525
  • Plot improvements for TC #521
  • Use cartopy_offlinedata #524

Bug Fixes

  • Fix post-processing induced warm bias over South Polar area #523
  • Fix flipped lat of ozone obs #491
  • Modify LH formula for NET_FLUX_SRF #519
  • Update remaining acme_diags/ refs to e3sm_diags/ #530
  • Add xml support for get_static_var #516

Cleanup

  • Rename dir acme_diags to e3sm_diags #495
  • Consolidate pre-processing scripts #514

Docs

  • Update load e3sm unified paths #489
  • Update README #507
  • Update dev install docs #508
  • Update unified paths #510

CI/CD

  • Enable ability to manually update docs on gh-pages #497
  • Update CI/CD workflows to only regenerate master docs #498
  • Update publish-docs job in release_workflow.yml #500
  • Add tbump to simplify package versioning #494

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by chengzhuzhang over 3 years ago

E3SM Diagnostics Package - v2.6.0rc1

Bug Fixes

  • Fix flipped lat of ozone obs #491
  • Modify LH formula for NET_FLUX_SRF #519
  • Update remaining acme_diags/ refs to e3sm_diags/ #530

Features

  • Add xml support for get_static_var #516
  • Refine plevs to include stratosphere #525
  • Plot improvements for TC #521
  • Use cartopy_offlinedata #524

Docs

  • Update load e3sm unified paths #489
  • Update README #507
  • Update dev install docs #508
  • Update unified paths #510

CI/CD

  • Enable ability to manually update docs on gh-pages #497
  • Update CI/CD workflows to only regenerate master docs #498
  • Update publish-docs job in release_workflow.yml #500
  • Add tbump to simplify package versioning #494

Cleanup

  • Rename dir acme_diags to e3sm_diags #495
  • Consolidate pre-processing scripts #514

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by tomvothecoder over 3 years ago

E3SM Diagnostics Package - TC Analysis and Column Ozone

New features include:

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by chengzhuzhang almost 4 years ago

E3SM Diagnostics Package - v2.5.0rc4

New features include:

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by forsyth2 almost 4 years ago

E3SM Diagnostics Package - v2.5.0rc3

New features include:

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by chengzhuzhang almost 4 years ago

E3SM Diagnostics Package - v2.5.0rc2

New features include:

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by forsyth2 almost 4 years ago

E3SM Diagnostics Package - TC Analysis and Zonal Mean Annual Cycle

New features include:

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by chengzhuzhang almost 4 years ago

E3SM Diagnostics Package - ARM Diags

This is a major release.

New Features:

  • Incorporation of ARM-diags which uses in-situ measurements from DOE's ARM (Atmospheric Radiation Measurement) Program to compare with point grid model output. Diagnostics added include: seasonal mean, annual cycle cycle of a set of atmospheric variables, including (surface temperature, precipitation, radiation fluxes, surface turbulent fluxes...); diurnal cycle of precipitation and cloud; convection onset metrics.

  • MERRA 2 datasets are updated to include monthly main daily maximum and minimum surface air temperature.

Other major enhancement

  • Add a test suite for E3SM-Diags on Chrysalis/Anvil
  • Image comparison tests added in unit tests and a complete run
  • Integrate GitHub Actions CI/CD workflow
  • Add and apply formatters and linters to codebase

Minor enhancement

  • Code repo clean up by moving test data to E3SM data server and providing data download script.
  • Simplify Conda environments

Documentation

  • Include quick guide for Chrysalis/Anvil
  • Update post-processing and run scripts examples
  • Update development guide

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by chengzhuzhang about 4 years ago

E3SM Diagnostics Package - v2.3.0

  • New set: streamflow diags including seasonality maps, annual mean maps and scatter plots based on gauge station data from GSIM
  • New set: diurnal cycle of precipitation diagnostics based on 3hourly TRMM data.
  • Updated documentation and bug fixes

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by chengzhuzhang over 4 years ago

E3SM Diagnostics Package - v2.2.0

  • Add 2020 tutorial examples
  • Add ERA5 and 1 deg GPCP data
  • Removed vtk as a dependency
  • Fixes for plotting issues 330 326

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by chengzhuzhang over 4 years ago

E3SM Diagnostics Package - v2.1.1

A minor release to address an issue likely stems from cartopy0.18 #318
As a workaround, polar plots are now in square shape.

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by chengzhuzhang almost 5 years ago

E3SM Diagnostics Package - v2.1.0

New Features:

  • Two components of ENSO-diags:
    maps of regression coefficient of atmospheric fields over SST anomaly;
    scatter plots of atmospheric feedback on SST anomaly.

  • Quasi-biennial Oscillation (QBO) analysis

  • Initial support for lat-lon map for Regional Refined Model (RRM) data.

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by chengzhuzhang almost 5 years ago

E3SM Diagnostics Package - v2.0.0

Some major changes in this new release:

  • Refactored the code structure to make the code base more flexible to incorporate new diagnostics sets.
  • Introduced a new way of running: run with API (i.e. python run_e3sm_diags.py). The old way (e3sm_diags -p) is backward compatible for the core diagnostics sets.
  • Added time-series diagnostics for annual mean trend for selected variables (can be ran only with the new way)
  • Variables P-E from (GPCP and OAflux), and from Core v2 fluxes are added.
  • Dataset class is refactored to be cleaner and more flexible.
  • Documentation is updated to reflect the new changes.

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by chengzhuzhang over 5 years ago

E3SM Diagnostics Package - v1.7.1

  • Add OMEGA to lat lon diagnostics set
  • Fix Dockerfile
  • Other bug fixes

We will have a big release for a new way of running the package, including adding set-specific parameters (i.g., plevs for the zonal_mean_2d diags set).
Please stay tuned!

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by chengzhuzhang almost 6 years ago

E3SM Diagnostics Package - v1.7.0

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by chengzhuzhang almost 6 years ago

E3SM Diagnostics Package - v1.6.1

  • No longer pinning specific versions of CDAT.
    • So using >= instead of =.
  • Builds are now noarch.

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by zshaheen about 6 years ago

E3SM Diagnostics Package - v1.6.0

  • In addition to climatology files, e3sm_diags now supports time series input (monthly data of multiple years saved in one file per each variable) to have climatology computed during runtime.
  • New quick start guide for NERSC Cori.
  • Major code refactor for reading in data and getting variables.
  • Scripts added for creating the time series and climatology files for the observational data.
  • User-facing instances of "acme" are replaced with "e3sm".
    • Please don't change the project name again.

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by zshaheen over 6 years ago

E3SM Diagnostics Package - v1.5.0

  • e3sm_diags can now be ran as a container, in addition to being manually installed and ran that way.
    • View how to do so on Edison at NERSC via Shifter here.
    • The auxiliary script to run a container,e3sm_diags_container.py, supports Docker and Shifter.
    • Singularity support will be added in the next release or so.
    • A quick guide will be added for Docker and Singularity soon as well.
  • Scripts added for creating the time-series observational data.
  • Minor additions to the viewer.

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by zshaheen over 6 years ago

E3SM Diagnostics Package - v1.4.0

  • Added selectors parameter.
    • View the documentation here.
  • Major bug-fix regarding issue #156, where too many threads were used.
  • Added scripts to create climo and timeseries data for the observational data. Located in the analysis_data_preprocess folder.

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by zshaheen over 6 years ago

E3SM Diagnostics Package - v.1.3.4

  • Fixed major issue where NetCDF-4 files would not open on Edison compute nodes.
  • Added output_format_subplot parameter to save individual panels when using matplotlib for the plotsets:
    • lat_lon
    • polar
    • zonal_mean_xy
    • zonal_mean_2d
    • cosp_histogram
  • Expanded provenance. Now for each run, the command used, the py file, and the cfg file used are save in {results_dir}/prov. A yml file of the environment is also saved there as well.

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by zshaheen almost 7 years ago

E3SM Diagnostics Package - v1.3.3

  • Finally renamed package to e3sm_diags.
    • But acme_diags is still supported for those who update and don't read the changelogs.
  • Removed mesalib as a dependency.
  • Other very minor fixes.

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by zshaheen about 7 years ago

E3SM Diagnostics Package - v1.3.2

  • Python 2 and 3 support.

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by zshaheen about 7 years ago

E3SM Diagnostics Package - v1.3.1

  • CDAT 8 support.
  • Use conservative regridding by default.
  • Expanded provenance:
    • Dump of which dependencies are used, stored in results_dir/env_prov.txt.
    • All parameters work via the command line.
    • --no_viewer parameter to not create the viewer.

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by zshaheen about 7 years ago

E3SM Diagnostics Package - v1.3.0

  • Stronger support for model vs model run with a new parameter run_type
  • Cleaner code directory with all default configuration files converted to .cfg
  • Add AOD550, surface wind stress and surface air temperature as default variables
  • Several bug fixes

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by zshaheen about 7 years ago

E3SM Diagnostics Package - v1.2.1

  • The provenance works better. The command should only create one plot.
  • Running with the mpl backend on a headless machine should work.

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by zshaheen over 7 years ago

E3SM Diagnostics Package - v1.2.0

  • Added Taylor Diagrams for spatial variability for annual and seasonal means.
  • Added provenance: a single run can be recreated with a single command.
  • A dependency, cdp, was updated to support the provenance feature.
    • If using a development environment, explained here, create a new environment by following the instructions again.
    • If using a regular environment, just run conda update acme_diags to get v1.2.0 of acme_diags.

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by chengzhuzhang over 7 years ago

E3SM Diagnostics Package - v1.1.1

  • Implemented a workaround for multiprocess running with VCS as the backend

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by zshaheen over 7 years ago

E3SM Diagnostics Package - v1.1.0

  • Added table to summarize metrics for each season, as part of the lat-lon diagnostics set
  • Added new variables from observational and reanalysis datasets, including:
    • NetCF (CERES-EBAF TOA, surface)
    • SHFLX, netSW surface, netLW surface, net flux sfc (ERA-Interim)
    • LHFLX, SHFLX, netSW surface, netLW surface, net flux sfc (MERRA)
    • Sea level pressure
  • Improved subtitles for lat-lon maps to reflect the years the climatology was averaged over
  • Added a "short_test_name" parameter for user-defined model names

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by zshaheen over 7 years ago

E3SM Diagnostics Package - v1.0.1

  • When a variable isn't present or there's an error in the code for a given diagnostics run, that run is skipped and the other diagnostics runs are ran.
  • Expanded on the zonal mean line plots, latitude-longitude contour maps, and polar contour maps default diagnostics.
  • Added system tests in tests/all_sets.cfg, use acme_diags_driver.py -d all_sets.cfg to generate one of each plot type in under 1 minute to test that an environment is working correctly.

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by zshaheen over 7 years ago

E3SM Diagnostics Package - v1.0.0 Documentation

  • View the documentation for acme_diags v1.0.0 here.

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by zshaheen over 7 years ago

E3SM Diagnostics Package - v1.0.0

  • Support for diagnostics based on seasonal or annual climatology data, including:
    • Latitude-Longitude contour maps (AMWG set 5)
    • Polar contour maps (AMWG set 7)
    • Zonal mean line plots (AMWG set 3)
    • Pressure-Latitude zonal mean contour plots (AMWG set 4)
    • CloudTopHeight-Tau joint histograms (AMWG set 13).
  • Diagnostics for model vs obs, obs vs obs, and model vs model.
  • Updated observational datasets available on LLNL ACME1/AIMS4 and NERSC.
  • Two graphical backends: VCS and Matplotlib with cartopy.
  • User–addable diagnostics during runtime.
  • Diagnostics can be run in serial, or in parallel with multiprocessing or distributedly.
  • Documentation including quick start guides for LLNL ACME1/AIMS4 and NERSC, detailed user’s guide, and examples.

Climate Change - Earth and Climate Modeling - Jupyter Notebook
Published by zshaheen over 7 years ago