Data

Tools

Indexes

Applications

Experiments

Open Sustainable Technology

A curated list of open technology projects to sustain a stable climate, energy supply, biodiversity and natural resources.

Recent Releases of xscen

xscen - v0.12.1

Contributors to this version: Juliette Lavoie (@juliettelavoie), Gabriel Rondeau-Genesse (@RondeauG), Éric Dupuis (@coxipi), Pascal Bourgault (@aulemahal).

New features and enhancements

  • xscen officially supports Python 3.13. (PR/551).

Breaking changes

  • Make strip_cat_metadata False by default in xs.save_to_zarr and xs.save_to_netcdf. (PR/556, GH/555).
  • New official column driving_member. For RCMs, this should store the driver's realisation number, while the member column should now store the RCM simulation's realisation number, noted as "rX". This member should approximately map to the "realization" part of CORDEX-CMIP6's "version_realization" facet (the version part mapping to the already existing version column). The member restricting feature of search_data_catalogs has been adapted, but continues to work with catalogs missing the driving_member column. (PR/559).
  • Also adapted from the CORDEX-CMIP6 specifications, the driving_model column does not need to indicate the driver's institution name anymore. (PR/559).
  • For Python 3.13 support, xscen now requires clisops>=0.16.1 and xsdba>=0.4.0. (PR/551).
  • Minimum required intake-esm has been updated to >=2025.2.3. (PR/551).
  • Temporarily pinned numcodecs to <0.16.0 for compatibility with zarr. (PR/571).

Bug fixes

  • Fixed the default for xs.utils.maybe_unstack. (PR/553).
  • Patch xsdba.units.convert_units_to with xclim.core.units.convert_units_to with context="infer" locally in xs.train and xs.adjust instead of using xc.core.settings.context. (PR/552).
  • Fixed a bug in xs.utils.clean_up where attributes would be dropped when using the missing_by_vars argument. (PR/569, GH/568).
  • Allow undetectable frequencies in xs.extract.resample. (PR/567).

Internal changes

  • Added the ability to test xESMF-related functions with tox / pip. (PR/554).
  • Updated the pins for xclim, xarray, dask, and rechunker. (PR/570).
  • More accurate listing of dependencies for the project in pyproject.toml and environment*.yml. (PR/557).
  • sphinx dependencies are more streamlined in the docs environment. (PR/557).
  • Added codespell, deptry, vulture, and yamllint to the linting checks. (PR/557).

Climate Change - Climate Data Processing and Analysis - Python
Published by Zeitsperre 21 days ago

xscen - v0.12.0

Contributors to this version: Trevor James Smith (@Zeitsperre), Pascal Bourgault (@aulemahal), Juliette Lavoie (@juliettelavoie), Sarah Gammon (@SarahG-579462), Éric Dupuis (@coxipi).

Breaking changes

  • xscen now uses flit as its build-engine and no longer uses setuptools, setuptools-scm, or wheel. (PR/519).
  • Update to support Python3.13 and xclim v0.55.0 (PR/532).
  • xscen now requires the xsdba package for bias adjustment functionality (replacement for xclim.sdba). (PR/530).

New features and enhancements

  • Include station-obs and forecasts in the derived schema for build_path. (PR/534).
  • Project catalog now allows check_valid and drop_duplicates keyword arguments. (PR/536, GH/535).
  • Add annual global tas timeseries for CMIP6's models CanESM5 r1i1p2f1 (ssp126, ssp245, ssp370, ssp585), MPI-ESM1-2-LR ssp370 (r2i1p1f1, r3i1p1f1, r4i1p1f1, r5i1p1f1) (PR/544).
  • Allow pd.Timestamp and more precise datetime strings for xs.search_data_catalogs and dc.search (PR/547, GH/546)

Bug fixes

  • Docstrings and documentation have been updated to remove several small grammatical errors. (PR/527).

Internal changes

  • Updated the cookiecutter template to the latest commit. (PR/527):
    • Updated versions of many GitHub Actions and Python dependencies.
    • Removed coveralls from the CI dependencies.
    • Added pre-commit hooks for vulture (dead code) and codespell (typos).
  • The minimum supported clisops version has been raised to v0.15.0. (PR/533).
  • Dependency pins have been synchronized across the repository. (PR/533).
  • GitHub Workflows for conda builds now use the coverallsapp/github-action action for coverage reporting. (PR/533).
  • xsdba is now used instead of xclim.sdba. (PR/530).

Climate Change - Climate Data Processing and Analysis - Python
Published by github-actions[bot] about 2 months ago

xscen - v0.11.0

Contributors to this version: Gabriel Rondeau-Genesse (@RondeauG), Juliette Lavoie (@juliettelavoie), Trevor James Smith (@Zeitsperre).

New features and enhancements

  • Improve xs.ensembles.build_partition_data. (PR/504).

Breaking changes

  • xs.utils.publish_release_notes and xs.utils.show_versions have been moved to xs.testing. (PR/492).
  • The previously-deprecated xs.reduce module has been removed. Refer to xs.ensembles.make_criteria and xs.ensembles.reduce_ensemble for replacement functionality. (PR/517).

Bug fixes

  • Added a missing library (openpyxl) to the requirements. (PR/492).
  • Fixed a bug in xs.io.subset_maxsize where the function would drop the last year. (PR/492).
  • Fixed a bug in xs.io.clean_incomplete where the .zmetadata file was not removed. (PR/492).
  • Fixed a bug in the saving of datasets where encoding was sometimes not applied, resulting for example in rechunking not being respected. (PR/492).
  • Fixed multiple bugs in xs.io.save_to_zarr with mode='a'. (PR/492).
  • Fixed a few minor bugs in xs.io.save_to_table. (PR/492).

Internal changes

  • Added a new parameter latest to xs.testing.publish_release_notes to only print the latest release notes. (PR/492).
  • The estimation method in xs.io.estimate_chunks has been improved. (PR/492).
  • A new parameter incomplete has been added to xs.io.clean_incomplete to remove incomplete variables. (PR/492).
  • Continued work on adding tests. (PR/492).
  • Modified a CI build to test against the oldest supported version of xclim. (PR/505).
  • Updated the cookiecutter template version: (PR/507)
    • Added vulture to pre-commit hooks (finding dead code blocks).
    • Added zizmor to the pre-commit hooks (security analysis for CI workflows).
    • Secured token usages on all workflows (using zizmor).
    • Simplified logic in bump-version.yml.
    • Synchronized a few dependencies.
  • Fixed a few socket blocks and configuration issues in the CI workflows. (PR/512).
  • Added Open Source Security Foundation Best Practices badge, Zenodo DOI badge, FOSSA license compliance badge to the README. (PR/514).
  • Several deprecated usages within the code base have been addressed. The number of warnings emitted from the test suite have been significantly reduced. (GH/515, PR/516).

Climate Change - Climate Data Processing and Analysis - Python
Published by github-actions[bot] 3 months ago

xscen - v0.10.1

v0.10.1 (2024-11-04)

Contributors to this version: Gabriel Rondeau-Genesse (@RondeauG), Pascal Bourgault (@aulemahal), Éric Dupuis (@coxipi), Juliette Lavoie (@juliettelavoie).

New features and enhancements

  • xs.io.make_toc now includes the global attributes of the dataset after the information about the variables. (PR/473).
  • New function xs.get_warming_level_from_period to get the warming level associated with a given time horizon. (PR/474).
  • Added ability to skip whole folders to xs.parse_directory with argument skip_dirs. (PR/478, PR/479).
  • diagnostics.measures_improvement now accepts dim, which specifies dimension(s) on which the proportion of improved pixels are computed. (PR/416)
  • The argument indicators in xs.produce_horizon is now optional. Added an argument op to control the climatological operation. (PR/483).

Breaking changes

  • xs.get_warming_level has been renamed to xs.get_period_from_warming_level. Its argument return_horizon was reversed and renamed return_central_year (PR/474).
  • Removed support for the deprecated xclim function change_significance in ensemble_stats. (PR/482).
  • The argument indicators in xs.produce_horizon is no longer positional. (PR/483).

Bug fixes

  • xs.io.save_to_table now correctly handles the case where the input is a DataArray or a Dataset with a single variable. (PR/473).
  • Fixed a bug in xs.utils.change_units where the original dataset was also getting modified. (PR/482).
  • Fixed a bug in xs.compute_indicators where the cat:variable attribute was not correctly set. (PR/483).
  • Fixed a bug in xs.climatological_op where kwargs were not passed to the operation function. (PR/486).
  • Fixed a bug in xs.climatological_op where min_periods was not passed when the operation was linregress. (PR/486).

Internal changes

  • Include CF convention for temperature differences and on scale (PR/428, GH/428).
  • Bumped the version of xclim to 0.53.2. (PR/482).
  • More tests added. (PR/486).
  • Fixed a bug in xs.testing.datablock_3d where some attributes of the rotated pole got reversed half-way through the creation of the dataset. (PR/486).
  • The function xs.regrid._get_grid_mapping was moved to xs.spatial.get_grid_mapping and is now a public function. (PR/486).

Climate Change - Climate Data Processing and Analysis - Python
Published by RondeauG 6 months ago

xscen - v0.10.0

v0.10.0 (2024-09-30)

Contributors to this version: Juliette Lavoie (@juliettelavoie), Pascal Bourgault (@aulemahal), Gabriel Rondeau-Genesse (@RondeauG), Trevor James Smith (@Zeitsperre).

New features and enhancements

  • The mask argument in stack_drop_nans can now be a list of dimensions. In that case, a dropna(how='all') operation will be used to create the mask on-the-fly. (PR/450).
  • Few changes to clean_up:
    • The convert_calendar function now uses xarray instead of xclim. (PR/450).
    • The attrs_to_remove and remove_all_attrs_except arguments now use real regex. (PR/450).
    • Multiple entries can now be given for change_attr_prefix. (PR/450).
  • minimum_calendar now accepts a list as input. (PR/450).
  • More calendars are now recognized in translate_time_chunk. (PR/450).
  • new_dim in unstack_dates is now None by default and changes depending on the frequency. It becomes month if the data is exactly monthly, and keep the old default of season otherwise. (PR/450).
  • Updated the list of libraries in show_versions to reflect our current environment. (PR/450).
  • New xscen.catutils.patterns_from_schema to generate all possible patterns from a given schema (or one of xscen's default), to use with :py:func:parse_directory. (PR/431).
  • New DataCatalog.copy_files to copy all files of catalog to a new destination, unzipping if needed and returning a new catalog. (PR/431).
  • Convenience functions xs.io.zip_directory and xs.io.unzip_directory (for zarrs). (PR/431).
  • New argument compute_indicators: rechunk_input to rechunk the inputs to resample-appropriate chunks before calling xclim. (PR/431).
  • New xs.indicators.get_indicator_outputs to retrieve what variable name(s) and frequency to expect from an xclim indicator. (PR/431).
  • xscen now supports launches tests from pytest with the --numprocesses option. See the pytest-xdist documentation <https://pytest-xdist.readthedocs.io/en/stable/>_ for more information. (PR/464).
  • Conservative regridding now supports oblique mercator projections. (PR/467).
  • The automatic name for the weight file in regrid_dataset is now more explicit to avoid errors, but now requires cat:id and cat:domain arguments for both the source and target datasets. (PR/467).

Bug fixes

  • Fixed bug with reusing weights. (GH/411, PR/414).
  • Fixed bug in update_from_ds when "time" is a coordinate, but not a dimension. (:pull: 417).
  • Avoid modification of mutable arguments in search_data_catalogs (PR/413).
  • ensure_correct_time now correctly handles cases where timesteps are missing. (PR/440).
  • If using the argument tile_buffer with a shape method in spatial.subset, the shapefile will now be reprojected to a WGS84 grid before the buffer is applied. (PR/440).
  • maybe_unstack now works if the dimension name is not the default. (PR/450).
  • unstack_fill_nan now works if given a dictionary that contains both dimensions and coordinates. (PR/450).
  • clean_up no longer modifies the original dataset. (PR/450).
  • unstack_dates now works correctly for yearly datasets when winter_starts_year=True, as well as multi-year datasets. (PR/450).
  • Fix xs.catalog.concat_data_catalogs for catalogs that have not been search yet. (PR/431).
  • Fix indicator computation using freq=2Q* by assuming this means a semiannual frequency anchored at the given month (pandas assumes 2 quarter steps, any of them anchored at the given month). (PR/431).
  • create_bounds_rotated_pole now uses the default value if the dataset has no north_pole_grid_longitude attribute, instead of crashing. (PR/455).
  • Rewrote the global tas data file with latest HDF5/h5py to avoid errors when using h5py 3.11 and hdf5 1.14.2. (PR/1861).
  • Remove reference of deprecated xclim functions (convert_calendar, get_calendar) and adapt the code for supporting xclim 0.52.2 and its subsequent development version. (PR/465).

Breaking changes

  • convert_calendar in clean_up now uses xarray instead of xclim. Keywords aren't compatible between the two, but given that xclim will abandon its function, no backwards compatibility was sought. (PR/450).
  • attrs_to_remove and remove_all_attrs_except in clean_up now use real regex. It should not be too breaking since a fullmatch() is used, but * is now .*. (PR/450).
  • Python 3.9 is no longer supported. (PR/456).
  • Functions and arguments that were deprecated in xscen v0.8.0 or earlier have been removed. (PR/461).
  • pytest-xdist is now a development dependency. (PR/464).
  • xs.regrid.create_bounds_rotated_pole has been renamed to xs.regrid.create_bounds_gridmapping. (PR/467).
  • The weights_location argument in regrid_dataset is no longer positional. (PR/467).
  • The xs.regrid.create_mask function now requires explicit arguments instead of a dictionary. (PR/467).

Internal changes

  • DataCatalog.to_dataset can now accept a preprocess argument even if create_ensemble_on is given. The user assumes calendar handling. (PR/431).
  • Include domain in weight_location in regrid_dataset. (PR/414).
  • Added pins to xarray, xclim, h5py, and netcdf4. (PR/414).
  • Add .zip and .zarr.zip as possible file extensions for Zarr datasets. (PR/426).
  • Explicitly assign coords of multiindex in xs.unstack_fill_nan. (PR/427).
  • French translations are compiled offline. A new check ensures no PR are merged with missing messages. (GH/342, PR/443).
  • Continued work to add tests. (PR/450).
  • Updated the cookiecutter template via cruft: (PR/452)
    • GitHub Workflows that use rely on PyPI-based dependencies now use commit hashes.
    • Dependabot will now group updates by type.
    • Dependencies have been updated and synchronized.
    • Contributor guidance documentation has been adjusted.
    • numpydoc-validate has been added to the linting tools.
    • Linting checks are more reliant on ruff suggestions and stricter.
    • flake8-alphabetize has been replaced by ruff.
    • License information has been updated in the library top-level __init__.py.
  • Docstrings have been adjusted to meet the numpydoc standard. (PR/452).

CI changes

  • The bump-version.yml workflow now uses the Ouranosinc GitHub Helper Bot to sign bump version commits. (PR/462).

Climate Change - Climate Data Processing and Analysis - Python
Published by github-actions[bot] 7 months ago

xscen - v0.9.1

v0.9.1 (2024-06-04)

Contributors to this version: Pascal Bourgault (@aulemahal), Trevor James Smith (@Zeitsperre), Juliette Lavoie (@juliettelavoie).

Breaking changes

Bug fixes

  • Fixed defaults for xr_combine_kwargs in extract_dataset (PR/402).
  • Fixed bug with xs.utils.update_attr(GH/404, PR/405).
  • Fixed template 1 bugs due to changes in dependencies. ( PR/405).

Internal changes

  • cartopy has been pinned above version '0.23.0' in order to address a licensing issue. (PR/403).
  • The cookiecutter template has been updated to the latest commit via cruft. (PR/407).
    • GitHub Workflows now point to commits rather than tags.
    • Dependabot will now only update on a monthly schedule.
    • Dependencies have been updated and synchronized.
    • CHANGES.rst is now CHANGELOG.rst (see: KeepAChangelog).
    • The CODE_OF_CONDUCT.rst file adapted to Contributor Covenant v2.1 <https://www.contributor-covenant.org/version/2/1/code_of_conduct/>_.
    • Maintainer-specific directions are now found under releasing.rst

Climate Change - Climate Data Processing and Analysis - Python
Published by RondeauG 11 months ago

xscen - v0.9.0

v0.9.0 (2024-05-07)

Contributors to this version: Trevor James Smith (@Zeitsperre), Pascal Bourgault (@aulemahal), Gabriel Rondeau-Genesse (@RondeauG), Juliette Lavoie (@juliettelavoie), Marco Braun (@vindelico).

New features and enhancements

  • xs.reduce_ensemble will now call xclim.ensembles.create_ensemble and xclim.ensembles.make_critera if required. (PR/386).

Breaking changes

  • Removed support for the old instances of the region argument in spatial_mean, extract_dataset, and subset. (PR/367).
  • Removed xscen.extract.clisops_subset. (PR/367).
  • dtr (the function) was renamed to dtr_from_minmax to avoid confusion with the dtr variable. (PR/372).
  • The xscen.reduce module has been abandoned. (PR/386).
    • build_reduction_data has been made redundant by xclim.ensembles.make_critera and will be removed in a future release.
    • xscen.reduce.reduce_ensemble has been moved to xscen.ensembles.reduce_ensemble, as a module was no longer necessary.

Internal changes

  • Modified xscen.utils.change_unit to always adopt the name from the variables_and_units dictionary if the physical units are equal but their names are not (ex. degC <-> ˚C) (PR/373).
  • Updated the cookiecutter template to the latest version. (PR/358):
    • Addresses a handful of misconfigurations in the GitHub Workflows.
    • Added a few free grep-based hooks for finding unwanted artifacts in the code base.
    • Updated ruff to v0.2.0 and black to v24.2.0.
  • Added more tests. (PR/366, PR/367, PR/372).
  • Refactored xs.spatial.subset into smaller functions. (PR/367).
  • An encoding argument was added to xs.config.load_config. (PR/370).
  • Various small fixes to the code to address FutureWarnings. (PR/380).
  • xs.spatial.subset will try to guess CF coordinate if it can't find "latitude" or "longitude" in ds.cf. (PR/384).
  • xs.extract_dataset and xs.DataCatalog.to_dataset will now default to opening datasets with option chunks={}, which tries to respect chunking on disk. (PR/398, GH/368).

Bug fixes

  • Fix unstack_dates for the new frequency syntax introduced by pandas v2.2. (PR/359).
  • subset_warming_level will not return partial subsets if the warming level is reached at the end of the timeseries. (GH/360, PR/359).
  • Loading of training in adjust is now done outside of the periods loop. (PR/366).
  • Fixed bug for adding the preprocessing attributes inside the adjust function. (PR/366).
  • Fixed a bug to accept group = False in adjust function. (PR/366).
  • creep_weights now correctly handles the case where the grid is small, n is large, and mode=wrap. (GH/367).
  • Fixed a bug in tasmin_from_dtr and tasmax_from_dtr, when dtr units differed from tasmin/max. (PR/372).
  • Fixed a bug where the requested chunking would be ignored when saving a dataset (PR/379).
  • The missing value check in health_checks will no longer crasg if a variable has no time dimension. (PR/382).

Climate Change - Climate Data Processing and Analysis - Python
Published by github-actions[bot] 12 months ago

xscen - v0.8.3

Contributors to this version: Juliette Lavoie (@juliettelavoie), Trevor James Smith (@Zeitsperre), Gabriel Rondeau-Genesse (@RondeauG), Pascal Bourgault (@aulemahal).

Announcements

Internal changes

  • Added tests for diagnostics. (PR/352).
  • Added a SECURITY.md file to the repository and the documentation. (PR/353).
  • Added tox modifier for testing builds against the main development branch of xclim. (PR/351, PR/355).
  • Added a requirements_upstream.txt file to the repository to track the development branches of relevant dependencies. (PR/355).
  • Added a dedicated GitHub Workflow to evaluate compatibility with upstream dependencies. (PR/355).

Breaking changes

  • xscen now requires pandas >= 2.2 and xclim >= 0.48.2. (PR/351).
  • Functions that output a dict with keys as xrfreq (such as extract_dataset, compute_indicators) will now return the new nomenclature (e.g. "YS-JAN" instead of "AS-JAN"). (PR/351).
  • Going from xrfreq to frequencies or timedeltas will still work, but the opposite (frequency --> xrfreq/timedelta) will now only result in the new pandas nomenclature. (PR/351).

Climate Change - Climate Data Processing and Analysis - Python
Published by Zeitsperre about 1 year ago

xscen - v0.8.2

v0.8.2 (2024-02-12)

Contributors to this version: Trevor James Smith (@Zeitsperre), Pascal Bourgault (@aulemahal)

New features and enhancements

  • Added a new argument indicators_kw to xs.ensembles.build_partition_data. (PR/315).
  • xscen is Semantic Versioning 2.0.0 <https://semver.org/spec/v2.0.0.html>_ compliant. (PR/319).
  • xesmf made an optional dependency, making xscen easier to install with pip. (PR/337).

Internal changes

  • Granular permissions and dependency scanning actions have been added to all GitHub CI Workflows. (PR/313).
  • Updated the list of dependencies to add missing requirements. (PR/314).
  • The cookiecutter template has been updated to the latest commit via cruft. (PR/319):
    • actions-versions-updater.yml has been replaced with Dependabot (it's just better).
    • The OpenSSF scorecard.yml workflow has been added to the GitHub workflows to evaluate package security.
    • Code formatting tools (black, blackdoc, isort) are now hard-pinned. These need to be kept in sync with changes from pre-commit. (Dependabot should perform this task automatically.)
    • The versioning system has been updated to follow the Semantic Versioning 2.0.0 standard.

Climate Change - Climate Data Processing and Analysis - Python
Published by RondeauG about 1 year ago

xscen - v0.8.0

Changelog

v0.8.0 (2024-01-16)

Contributors to this version: Gabriel Rondeau-Genesse (@RondeauG), Pascal Bourgault (@aulemahal), Juliette Lavoie (@juliettelavoie), Sarah-Claude Bourdeau-Goulet (@sarahclaude), Trevor James Smith (@Zeitsperre), Marco Braun (@vindelico).

Announcements

  • xscen now adheres to PEPs 517/518/621 using the setuptools and setuptools-scm backend for building and packaging. (PR/292).

New features and enhancements

  • New function xscen.indicators.select_inds_for_avail_vars to filter the indicators that can be calculated with the variables available in a xarray.Dataset. (PR/291).
  • Replaced aggregation function climatological_mean() with climatological_op() offering more types of operations to aggregate over climatological periods. (PR/290)
  • Added the ability to search for simulations that reach a given warming level. (PR/251).
  • xs.spatial_mean now accepts the region="global" keyword to perform a global average (GH/94, PR/260).
  • xs.spatial_mean with method='xESMF' will also automatically segmentize polygons (down to a 1° resolution) to ensure a correct average (PR/260).
  • Added documentation for require_all_on in search_data_catalogs. (PR/263).
  • xs.save_to_table and xs.io.to_table to transform datasets and arrays to DataFrames, but with support for multi-columns, multi-sheets and localized table of content generation.
  • Better xs.extract.resample : support for weighted resampling operations when starting with frequencies coarser than daily and missing timesteps/values handling. (GH/80, GH/93, PR/265).
  • New argument attribute_weights to generate_weights to allow for custom weights. (PR/252).
  • xs.io.round_bits to round floating point variable up to a number of bits, allowing for a better compression. This can be combined with the saving step through argument "bitround" of save_to_netcdf and save_to_zarr. (PR/266).
  • Added annual global tas timeseries for CMIP6's models CMCC-ESM2 (ssp245, ssp370, ssp585), EC-Earth3-CC (ssp245, ssp585), KACE-1-0-G (ssp245, ssp370, ssp585) and TaiESM1 (ssp245, ssp370). Moved global tas database to a netCDF file. (GH/268, PR/270).
  • Implemented support for multiple levels and models in xs.subset_warming_level. Better support for DataArray and DataFrame in xs.get_warming_level. (PR/270).
  • Added the ability to directly provide an ensemble dataset to xs.ensemble_stats. (PR/299).
  • Added support in xs.ensemble_stats for the new robustness-related functions available in xclim. (PR/299).
  • New function xs.ensembles.get_partition_input (PR/289).

Breaking changes

  • climatological_mean() has been replaced with climatological_op() and will be abandoned in a future version. (PR/290)
  • experiment_weights argument in generate_weights was renamed to balance_experiments. (PR/252).
  • New argument attribute_weights to generate_weights to allow for custom weights. (PR/252).
  • For a sequence of models, the output of xs.get_warming_level is now a list. Revert to a dictionary with output='selected' (PR/270).
  • The global average temperature database is now a netCDF, custom databases must follow the same format (PR/270).

Bug fixes

  • Fixed a bug in xs.search_data_catalogs when searching for fixed fields and specific experiments/members. (PR/251).
  • Fixed a bug in the documentation build configuration that prevented stable/latest and tagged documentation builds from resolving on ReadTheDocs. (PR/256).
  • Fixed get_warming_level to avoid incomplete matches. (PR/269).
  • search_data_catalogs now eliminates anything that matches any entry in exclusions. (GH/275, PR/280).
  • Fixed a bug in xs.scripting.save_and_update where build_path_kwargs was ignored when trying to guess the file format. (PR/282).
  • Add a warning to xs.extract._dispatch_historical_to_future. (GH/286, PR/287).
  • Modify use_cftime for the calendar conversion in to_dataset. (GH/303, PR/289).

Internal changes

  • Continued work on adding tests. (PR/251).
  • Fixed pre-commit's pretty-format-json hook so that it ignores notebooks. (PR/254).
  • Fixed the labeler so docs/CI isn't automatically added for contributions by new collaborators. (PR/254).
  • Made it so that tests are no longer treated as an installable package. (PR/248).
  • Renamed the pytest marker from requires_docs to requires_netcdf. (PR/248).
  • Included the documentation in the source distribution, while excluding the NetCDF files. (PR/248).
  • Reduced the size of the files in /docs/notebooks/samples and changed the notebooks and tests accordingly. (GH/247, PR/248).
  • Added a new xscen.testing module with the datablock_3d function previously located in /tests/conftest.py. (PR/248).
  • New function xscen.testing.fake_data to generate fake data for testing. (PR/248).
  • xESMF 0.8 Regridder and SpatialAverager argument out_chunks is now accepted by xs.regrid_dataset and xs.spatial_mean. (PR/260).
  • Testing, Packaging, and CI adjustments. (PR/274):
    • xscen builds now install in a tox environment with conda-provided ESMF in GitHub Workflows.
    • tox now offers a method for installing esmpy from a tag/branch (via ESMF_VERSION environment variable).
    • $ make translate is now called on ReadTheDocs and within tox.
    • Linters are now called by order of most common failures first, to speed up the CI.
    • Manifest.in is much more specific about what is installed.
    • Re-adds a dev recipe to the setup.py.
  • Multiple improvements to the docstrings and type annotations. (PR/282).
  • pip check in conda builds in GitHub workflows have been temporarily set to always pass. (PR/288).
  • The cookiecutter template has been updated to the latest commit via cruft. (PR/292):
    • setup.py has been mostly hollowed-out, save for the babel-related translation function.
    • pyproject.toml has been added, with most package configurations migrated into it.
    • HISTORY.rst has been renamed to CHANGES.rst.
    • actions-version-updater.yml has been added to automate the versioning of the package.
    • pre-commit hooks have been updated to the latest versions; check-toml and toml-sort have been added to cleanup the pyproject.toml file, and check-json-schema has been added to ensure GitHub and ReadTheDocs workflow files are valid.
    • ruff has been added to the linting tools to replace most flake8 and pydocstyle verifications.
    • tox builds are more pure Python environment/PyPI-friendly.
    • xscen now uses Trusted Publishing for TestPyPI and PyPI uploads.
  • Linting checks now examine the testing folder, function complexity, and alphabetical order of __all__ lists. (PR/292).
  • publish_release_notes now uses better logic for finding and reformatting the CHANGES.rst file. (PR/292).
  • bump2version version-bumping utility was replaced by bump-my-version. (PR/292).
  • Documentation build checks no longer fail due to broken external links; Notebooks are now nested and numbered. (PR/304).

Climate Change - Climate Data Processing and Analysis - Python
Published by github-actions[bot] over 1 year ago

xscen - v0.7.1

Bug fix release to update the dependencies and clean the way for a working conda-forge package!

Also, a bug fix in aggregate.spatial_mean for method= 'cos-lat' and the case of a rectilinear grid.

Climate Change - Climate Data Processing and Analysis - Python
Published by aulemahal over 1 year ago

xscen - v0.7.0

v0.7.0 (2023-08-22)

Contributors to this version: Gabriel Rondeau-Genesse (@RondeauG), Pascal Bourgault (@aulemahal), Trevor James Smith (@Zeitsperre), Juliette Lavoie (@juliettelavoie), Marco Braun (@vindelico).

Announcements

  • Dropped support for Python 3.8, added support for 3.11. (PR/199, PR/222).
  • xscen is now available on conda-forge <https://anaconda.org/conda-forge/xscen>_, and can be installed with conda install -c conda-forge xscen. (PR/241)

New features and enhancements

  • xscen now tracks code coverage using coveralls. (PR/187).
  • New function get_warming_level to search within the IPCC CMIP global temperatures CSV without requiring data. (GH/208, PR/210).
  • File re-structuration from catalogs with xscen.catutils.build_path. (PR/205, PR/237).
  • New scripting functions save_and_update and move_and_delete. (PR/214).
  • Spatial dimensions can be generalized as X/Y when rechunking and will be mapped to rlon/rlat or lon/lat accordingly. (PR/221).
  • New argument var_as_string for get_cat_attrs to return variable names as strings. (PR/233).
  • New argument copy for move_and_delete. (PR/233).
  • New argument restrict_year for compute_indicators. (PR/233).
  • Add more comments in the template. (PR/233, GH/232).
  • generate_weights now allows to split weights between experiments, and make them vary along the time/horizon axis. (GH/108, PR/231).
  • New independence_level, institution, added to generate_weights. (PR/231).
  • Updated produce_horizon so it can accept multiple periods or warming levels. (PR/231, PR/240).
  • Add more comments in the template. (PR/233, PR/235, GH/232).
  • New function diagnostics.health_checks that can perform multiple checkups on a dataset. (PR/238).

Breaking changes

  • Columns date_start and date_end now use a datetime64[ms] dtype. (PR/222).
  • The default output of date_parser is now pd.Timestamp (output_dtype='datetime'). (PR/222).
  • date_parser(date, end_of_period=True) has time "23:59:59", instead of "23:00". (PR/222, PR/237).
  • driving_institution was removed from the "default" xscen columns. (PR/222).
  • Folder parsing utilities (parse_directory) moved to xscen.catutils. Signature changed : globpattern removed, dirglob added, new patterns specifications. See doc for all changes. (PR/205).
  • compute_indicators now returns all outputs produced by indicators with multiple outputs (such as rain_season). (PR/228).
  • In generate_weights, independence_level all was renamed model. (PR/231).
  • In response to a bugfix, results for generate_weights(independence_level='GCM') are significantly altered. (GH/230, PR/231).
  • Legacy support for stats_kwargs in ensemble_stats was dropped. (PR/231).
  • period in produce_horizon has been deprecated and replaced with periods. (PR/231).
  • Some automated to_level were updated to reflect more recent changes. (PR/231).
  • Removed diagnostics.fix_unphysical_values. (PR/238).

Bug fixes

  • Fix bug in unstack_dates with seasonal climatological mean. (GH/202, PR/202).
  • Added NotImplemented errors when trying to call climatological_mean and compute_deltas with daily data. (PR/187).
  • Minor documentation fixes. (GH/223, PR/225).
  • Fixed a bug in unstack_dates where it failed for anything other than seasons. (PR/228).
  • cleanup with common_attrs_only now works even when no cat attribute is present in the datasets. (PR/231).

Internal changes

  • Removed the pin on xarray's version. (GH/175, PR/199).
  • Folder parsing utilities now in pure python, platform independent. New dependency parse. (PR/205).
  • Updated ReadTheDocs configuration to prevent --eager installation of xscen (PR/209).
  • Implemented a template to be used for unit tests. (PR/187).
  • Updated GitHub Actions to remove deprecation warnings. (PR/187).
  • Updated the cookiecutter used to generate boilerplate documentation and code via cruft. (PR/212).
  • A few changes to subset_warming_level so it doesn't need driving_institution. (PR/215).
  • Added more tests. (PR/228).
  • In compute_indicators, the logic to manage indicators returning multiple outputs was simplified. (PR/228).

Climate Change - Climate Data Processing and Analysis - Python
Published by RondeauG over 1 year ago

xscen - v0.6.0

History

v0.6.0 (2023-05-04)

Contributors to this version: Trevor James Smith (@Zeitsperre), Juliette Lavoie (@juliettelavoie), Pascal Bourgault (@aulemahal), Gabriel Rondeau-Genesse (@RondeauG).

Announcements

  • xscen is now offered as a conda package available through Anaconda.org. Refer to the installation documentation for more information. (GH/149, PR/171).
  • Deprecation: Release 0.6.0 of xscen will be the last version to support xscen.extract.clisops_subset. Use xscen.spatial.subset instead. (PR/182, PR/184).
  • Deprecation: The argument region, used in multiple functions, has been slightly reformatted. Release 0.6.0 of xscen will be the last version to support the old format. (GH/99, GH/101, PR/184).

New features and enhancements

  • New 'cos-lat' averaging in spatial_mean. (GH/94, PR/125).
  • Support for computing anomalies in compute_deltas. (PR/165).
  • Add function diagnostics.measures_improvement_2d. (PR/167).
  • Add function regrid.create_bounds_rotated_pole and automatic use in regrid_dataset and spatial_mean. This is temporary, while we wait for a functionning method in cf_xarray. (PR/174, GH/96).
  • Add spatial submodule with functions creep_weights and creep_fill for filling NaNs using neighbours. (PR/174).
  • Allow passing GeoDataFrame instances in spatial_mean's region argument, not only geospatial file paths. (PR/174).
  • Allow searching for periods in catalog.search. (GH/123, PR/170).
  • Allow searching and extracting multiple frequencies for a given variable. (GH/168, PR/170).
  • New masking feature in extract_dataset. (GH/180, PR/182).
  • New function xs.spatial.subset to replace xs.extract.clisops_subset and add method "sel". (GH/180, PR/182).
  • Add long_name attribute to diagnostics. ( PR/189).
  • Added a new YAML-centric notebook (GH/8, PR/191).
  • New utils.standardize_periods to standardize that argument across multiple functions. (GH/87, PR/192).
  • New coverage_kwargs argument added to search_data_catalogs to allow modifying the default values of subset_file_coverage. (GH/87, PR/192).

Breaking changes

  • 'mean' averaging has been deprecated in spatial_mean. (PR/125).
  • 'interp_coord' has been renamed to 'interp_centroid' in spatial_mean. (PR/125).
  • The 'datasets' dimension of the output of diagnostics.measures_heatmap is renamed 'realization'. (PR/167).
  • _subset_file_coverage was renamed subset_file_coverage and moved to catalog.py to prevent circular imports. (PR/170).
  • extract_dataset doesn't fail when a variable is in the dataset, but not variables_and_freqs. (PR/185).
  • The argument period, used in multiple function, is now always a single list, while periods is more flexible. (GH/87, PR/192).
  • The parameters reference_period and simulation_period of xscen.train and xscen.adjust were renamed period/periods to respect the point above. (GH/87, PR/192).

Bug fixes

  • Forbid pandas v1.5.3 in the environment files, as the linux conda build breaks the data catalog parser. (GH/161, PR/162).
  • Only return requested variables when using DataCatalog.to_dataset. (PR/163).
  • compute_indicators no longer crashes if less than 3 timesteps are produced. (PR/125).
  • xarray is temporarily pinned below v2023.3.0 due to an API-breaking change. (GH/175, PR/173).
  • `xscen.utils.unstack_fill_nan`` can now handle datasets that have non dimension coordinates. (GH/156, PR/175).
  • extract_dataset now skips a simulation way earlier if the frequency doesn't match. (PR/170).
  • extract_dataset now correctly tries to extract in reverse timedelta order. (PR/170).
  • compute_deltas no longer creates all NaN values if the input dataset is in a non-standard calendar. (PR/188).

Internal changes

  • xscen now manages packaging for PyPi and TestPyPI via GitHub workflows. (PR/159).
  • Pre-load coordinates in extract.clisops_subset (PR/163).
  • Minimal documentation for templates. (PR/163).
  • xscen is now indexed in Zenodo, under the ouranos community of projects. (PR/164).
  • Added a few relevant Shields to the README.rst. (PR/164).
  • Better warning messages in _subset_file_coverage when coverage is insufficient. (PR/125).
  • The top-level Makefile now includes a linkcheck recipe, and the ReadTheDocs configuration no longer reinstalls the llvmlite compiler library. (PR/173).
  • The checkups on coverage and duplicates can now be skipped in subset_file_coverage. (PR/170).
  • Changed the ProjectCatalog docstrings to make it more obvious that it needs to be created empty. (GH/99, PR/184).
  • Added parse_config to creep_fill, creep_weights, and reduce_ensemble (PR/191).

Climate Change - Climate Data Processing and Analysis - Python
Published by RondeauG almost 2 years ago

xscen - v0.5.0

Contributors to this version: Gabriel Rondeau-Genesse (@RondeauG), Juliette Lavoie (@juliettelavoie), Trevor James Smith (@Zeitsperre), Sarah Gammon (@sg2475962) and Pascal Bourgault (@aulemahal).

New features and enhancements

  • Possibility of excluding variables read from file from the catalog produced by parse_directory. (PR/107).
  • New functions extract.subset_warming_level and aggregate.produce_horizon. (PR/93).
  • add round_var to xs.clean_up. (PR/93).
  • New "timeout_cleanup" option for save_to_zarr, which removes variables that were in the process of being written when receiving a TimeoutException. (PR/106).
  • New scripting.skippable context, allowing the use of CTRL-C to skip code sections. (PR/106).
  • Possibility of fields with underscores in the patterns of parse_directory. (PR/111).
  • New utils.show_versions function for printing or writing to file the dependency versions of xscen. (GH/109, PR/112).
  • Added previously private notebooks to the documentation. (PR/108).
  • Notebooks are now tested using pytest with nbval. (PR/108).
  • New restrict_warming_level argument for extract.search_data_catalogs to filter dataset that are not in the warming level csv. (GH/105, PR/138).
  • Set configuration value programmatically through CONFIG.set. (PR/144).
  • New to_dataset method on DataCatalog. The same as to_dask, but exposing more aggregation options. (PR/147).

Breaking changes

  • Functions that are called internally can no longer parse the configuration. (PR/133).

Bug fixes

  • clean_up now converts the calendar of variables that use "interpolate" in "missing_by_var" at the same time.
    • Hence, when it is a conversion from a 360_day calendar, the random dates are the same for all of the these variables. (GH/102, PR/104).
  • properties_and_measures no longer casts month coordinates to string. (PR/106).
  • search_data_catalogs no longer crashes if it finds nothing. (GH/42, PR/92).
  • Prevented fixed fields from being duplicated during _dispatch_historical_to_future (GH/81, PR/92).
  • Added missing parse_config to functions in reduce.py (PR/92).
  • Added deepcopy before skipna is popped in spatial_mean (PR/92).
  • subset_warming_level now validates that the data exists in the dataset provided (GH/117, PR/119).
  • Adapt stack_drop_nan for the newest version of xarray (2022.12.0). (GH/122, PR/126).
  • Fix stack_drop_nan not working if intermediate directories don't exist (GH/128).
  • Fixed a crash when compute_indicators produced fixed fields (PR/139).

Internal changes

  • compute_deltas skips the unstacking step if there is no time dimension and cast object dimensions to string. (PR/9)
  • Added the "2sem" frequency to the translations CVs. (PR/111).
  • Skip files we can't read in parse_directory. (PR/111).
  • Fixed non-numpy-standard Docstrings. (PR/108).
  • Added more metadata to package description on PyPI. (PR/108).
  • Faster search_data_catalogs and extract_dataset through a faster DataCatalog.unique, date parsing and a rewrite of the ensure_correct_time logic. (PR/127).
  • The search_data_catalogs function now accepts str or pathlib.Path variables (in addition to lists of either data type) for performing catalog lookups. (PR/121).
  • produce_horizons now supports fixed fields (PR/139).
  • Rewrite of unstack_dates for better performance with dask arrays. (PR/144).

Climate Change - Climate Data Processing and Analysis - Python
Published by Zeitsperre about 2 years ago

xscen - v0.4.0

v0.4.0 (2022-09-28)

Contributors to this version: Gabriel Rondeau-Genesse (@RondeauG), Juliette Lavoie (@juliettelavoie), Trevor James Smith (@Zeitsperre) and Pascal Bourgault (@aulemahal).

New features and enhancements

  • New functions diagnostics.properties_and_measures, diagnostics.measures_heatmap and diagnostics.measures_improvement. (GH/5, PR/54).
  • Add argument resample_methods to xs.extract.resample. (GH/57, PR/57)
  • Added a ReadTheDocs configuration to expose public documentation. (GH/65, PR/66).
  • xs.utils.stack_drop_nans/ xs.utils.unstack_fill_nan will now format the to_file/coords string to add the domain and the shape. (GH/59, PR/67).
  • New unstack_dates function to "extract" seasons or months from a timeseries. (PR/68).
  • Better spatial_mean for cases using xESMF and a shapefile with multiple polygons. (PR/68).
  • Yet more changes to parse_directory: (PR/68).
    • Better parallelization by merging the finding and name-parsing step in the same dask tree.
    • Allow cvs for the variable columns.
    • Fix parsing the variable names from datasets.
    • Sort the variables in the tuples (for a more consistent output)
  • In extract_dataset, add option ensure_correct_time to ensure the time coordinate matches the expected freq. Ex: monthly values given on the 15th day are moved to the 1st, as expected when asking for "MS". (:issue: 53).
  • In regrid_dataset: (PR/68).
    • Allow passing skipna to the regridder kwargs.
    • Do not fail for any grid mapping problem, includin if a grid_mapping attribute mentions a variable that doesn't exist.
  • Default email sent to the local user. (PR/68).
  • Special accelerated pathway for parsing catalogs with all dates within the datetime64[ns] range. (PR/75).
  • New functions reduce_ensemble and build_reduction_data to support kkz and kmeans clustering. (GH/4, PR/63).
  • ensemble_stats can now loop through multiple statistics, support functions located in xclim.ensembles._robustness, and supports weighted realizations. (PR/63).
  • New function ensemble_stats.generate_weights that estimates weights based on simulation metadata. (PR/63).
  • New function catalog.unstack_id to reverse-engineer IDs. (PR/63).
  • generate_id now accepts Datasets. (PR/63).
  • Add rechunk option to properties_and_measures (PR/76).
  • Add create argument to ProjectCatalog (GH/11, PR/77).
  • Add percentage deltas to compute_deltas (GH/82, PR/90).

Breaking changes

  • statistics / stats_kwargs have been changed/eliminated in ensemble_stats, respectively. (PR/63).

Bug fixes

  • Add a missing dependencies to the env (pyarrow, for faster string handling in catalogs). (PR/68).
  • Allow passing compute=False to save_to_zarr. (PR/68).

Internal changes

  • Small bugfixes in aggregate.py. (PR/55, PR/56).
  • Default method of xs.extract.resample now depends on frequency. (GH/57, PR/58).
  • Bugfix for _restrict_by_resolution with CMIP6 datasets (PR/71).
  • More complete check of coverage in _subset_file_coverage. (GH/70, PR/72)
  • The code that performs common_attrs_only in ensemble_stats has been moved to clean_up. (PR/63).
  • Removed the default to_level in clean_up. (PR/63).
  • xscen now has an official logo. (PR/69).
  • Use numpy max and min in properties_and_measures (PR/76).
  • Cast catalog date_start and date_end to "%4Y-%m-%d %H:00" when writing to disk. (GH/83, PR/79)
  • Skip test of coverage on the sum if the list of select files is empty. (PR/79)
  • Added missing CMIP variable names in conversions.yml and added the ability to provide a custom file instead (GH/86, PR/88)
  • Changed 'allow_conversion' and 'allow_resample' default to False in search_data_catalogs (GH/86, PR/88)

Climate Change - Climate Data Processing and Analysis - Python
Published by RondeauG over 2 years ago

xscen - v0.3.0

Contributors to this version: Gabriel Rondeau-Genesse (@RondeauG), Juliette Lavoie (@juliettelavoie), Trevor James Smith (@Zeitsperre) and Pascal Bourgault (@aulemahal).

New features and enhancements

  • New function clean_up added. (GH/22, PR/24).
  • parse_directory: Fixes to xr_open_kwargs and support for wildcards (*) in the directories. (PR/19).
  • New function xscen.ensemble.ensemble_stats added. (GH/3, PR/28).
  • New functions spatial_mean, climatological_mean and deltas added. (GH/4, PR/35).
  • Add argument intermediate_reg_grids to xscen.regridding.regrid. (GH/34, PR/39).
  • Add argument moving_yearly_window to xscen.biasadjust.adjust. (PR/39).
  • Many adjustments to parse_directory: better wildcards (GH/24), allow custom columns, fastpaths for parse_from_ds, and more (PR/30).
  • Documentation now makes better use of autodoc to generate package index. (PR/41).
  • periods argument added to compute_indicators to support datasets with jumps in time (PR/35).

Breaking changes

  • Patterns in parse_directory start at the end of the paths in directories. (PR/30).
  • Argument extension of parse_directory has been renamed globpattern. (PR/30).
  • The xscen API and filestructure have been significantly refactored. (GH/40, PR/41). The following functions are available from the top-level:
    • adjust, train, ensemble_stats, clisops_subset, dispatch_historical_to_future, extract_dataset, resample, restrict_by_resolution, restrict_multimembers, search_data_catalogs, save_to_netcdf, save_to_zarr, rechunk, compute_indicators, regrid_dataset, and create_mask.
  • xscen now requires geopandas and shapely (PR/35).
  • Following a change in intake-esm xscen now uses "cat:" to prefix the dataset attributes extracted from the catalog. All catalog-generated attributes should now be valid when saving to netCDF. (GH/13, PR/51).

Internal changes

  • parse_directory: Fixes to xr_open_kwargs. (PR/19).
  • Fix for indicators removing the 'time' dimension. (PR/23).
  • Security scanning using CodeQL and GitHub Actions is now configured for the repository. (PR/21).
  • Bumpversion action now configured to automatically augment the version number on each merged pull request. (PR/21).
  • Add align_on = 'year' argument in bias adjustment converting of calendars. (PR/39).
  • GitHub Actions using Ubuntu-22.04 images are now configured for running testing ensemble using tox-conda. (PR/44).
  • import xscen smoke test is now run on all pull requests. (PR/44).
  • Fix for create_mask removing attributes (PR/35).

Climate Change - Climate Data Processing and Analysis - Python
Published by Zeitsperre over 2 years ago

xscen - v0.2.0

Contributors to this version: Gabriel Rondeau-Genesse (:user:RondeauG), Pascal Bourgault (:user:aulemahal), Trevor James Smith (:user:Zeitsperre), Juliette Lavoie (:user:juliettelavoie).

Announcements

  • This is the first official release for xscen!

New features and enhancements

  • Supports workflows with YAML configuration files for better transparency, reproducibility, and long-term backups.
  • Intake_esm-based catalog to find and manage climate data.
  • Climate dataset extraction, subsetting, and temporal aggregation.
  • Calculate missing variables through Intake-esm's DerivedVariableRegistry.
  • Regridding with xESMF.
  • Bias adjustment with xclim.

Breaking changes

  • N/A

Internal changes

  • N/A

Climate Change - Climate Data Processing and Analysis - Python
Published by RondeauG almost 3 years ago