Data

Tools

Indexes

Applications

Experiments

Open Sustainable Technology

A curated list of open technology projects to sustain a stable climate, energy supply, biodiversity and natural resources.

Recent Releases of pyaerocom

pyaerocom - v0.29.0

What's Changed

Full Changelog: https://github.com/metno/pyaerocom/compare/v0.28.0...v0.29.0

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by lewisblake 25 days ago

pyaerocom - v0.28.0

What's Changed

Full Changelog: https://github.com/metno/pyaerocom/compare/v0.27.0...v0.28.0

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by lewisblake about 2 months ago

pyaerocom - v0.27.0

What's Changed

Full Changelog: https://github.com/metno/pyaerocom/compare/v0.26.0...v0.27.0

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by lewisblake 3 months ago

pyaerocom - v0.26.0

What's Changed

New Contributors

Full Changelog: https://github.com/metno/pyaerocom/compare/v0.25.0...v0.26.0

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by lewisblake 4 months ago

pyaerocom - v0.25.0

What's Changed

New Contributors

Full Changelog: https://github.com/metno/pyaerocom/compare/v0.24.0...v0.25.0

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by lewisblake 5 months ago

pyaerocom - v0.24.0

What's Changed

Full Changelog: https://github.com/metno/pyaerocom/compare/v0.23.0...v0.24.0

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by lewisblake 6 months ago

pyaerocom - v0.23.0

What's Changed

Full Changelog: https://github.com/metno/pyaerocom/compare/v0.22.0...v0.23.0

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by lewisblake 7 months ago

pyaerocom - v0.22.0

What's Changed

Full Changelog: https://github.com/metno/pyaerocom/compare/v0.21.1...v0.22.0

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by heikoklein 8 months ago

pyaerocom - v0.21.1

Now has aerovaldb installed from PyPI

What's Changed

Full Changelog: https://github.com/metno/pyaerocom/compare/v0.21.0...v0.21.1

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by github-actions[bot] 9 months ago

pyaerocom - v0.21.0

What's Changed

Full Changelog: https://github.com/metno/pyaerocom/compare/v0.20.0...v0.21.0

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by lewisblake 9 months ago

pyaerocom - v0.20.0

What's Changed

Full Changelog: https://github.com/metno/pyaerocom/compare/v0.19.2...v0.20.0

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by lewisblake 10 months ago

pyaerocom - v0.19.2

What's Changed

Full Changelog: https://github.com/metno/pyaerocom/compare/v0.19.1...v0.19.2

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by lewisblake 11 months ago

pyaerocom - v0.19.1

What's Changed

Full Changelog: https://github.com/metno/pyaerocom/compare/v0.19.0...v0.19.1

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by lewisblake 11 months ago

pyaerocom - v0.19.0

What's Changed

Full Changelog: https://github.com/metno/pyaerocom/compare/v0.18.0...v0.19.0

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by lewisblake 11 months ago

pyaerocom - v0.18.0

What's Changed

Full Changelog: https://github.com/metno/pyaerocom/compare/v0.17.1...v0.18.0

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by lewisblake 12 months ago

pyaerocom - v0.17.1

What's Changed

Full Changelog: https://github.com/metno/pyaerocom/compare/v0.17.0...v0.17.1

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by lewisblake about 1 year ago

pyaerocom - v0.17.0

What's Changed

New Contributors

Full Changelog: https://github.com/metno/pyaerocom/compare/v0.16.0...v0.17.0

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by lewisblake about 1 year ago

pyaerocom - v0.16.0

What's Changed

New Contributors

Full Changelog: https://github.com/metno/pyaerocom/compare/v0.15.1...v0.16.0

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by lewisblake about 1 year ago

pyaerocom - v0.15.1post1

Update version number metadata to upload to PyPI

Full Changelog: https://github.com/metno/pyaerocom/compare/v0.15.1...v0.15.1post1

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by lewisblake about 1 year ago

pyaerocom - v0.15.1

Release for milestone m2024-02

What's Changed

Full Changelog: https://github.com/metno/pyaerocom/compare/v0.15.0...v0.15.1

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by lewisblake about 1 year ago

pyaerocom - v0.15.0

What's Changed

Full Changelog: https://github.com/metno/pyaerocom/compare/v0.14.0...v0.15.0

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by lewisblake over 1 year ago

pyaerocom - v0.14.0

What's Changed

Full Changelog: https://github.com/metno/pyaerocom/compare/v0.13.2...v0.14.0

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by github-actions[bot] over 1 year ago

pyaerocom - v0.13.2

pyaerocom v0.13.2 2022.11 release

What's Changed

Full Changelog: https://github.com/metno/pyaerocom/compare/v0.13.1...v0.13.2

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by github-actions[bot] over 2 years ago

pyaerocom - v0.13.1.post1

same as v0.13.1, including tests for conda-forge pipeline

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by github-actions[bot] almost 3 years ago

pyaerocom - v0.13.1

pyaerocom v0.13.1 2022.05 release

What's Changed

Full Changelog: https://github.com/metno/pyaerocom/compare/v0.13.0...v0.13.1

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by github-actions[bot] almost 3 years ago

pyaerocom - v0.13.0

pyaerocom v0.13.0 2022.05 release

What's Changed

New Contributors

Full Changelog: https://github.com/metno/pyaerocom/compare/v0.12.1...v0.13.0

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by github-actions[bot] almost 3 years ago

pyaerocom - Version 0.12.1

Same as v0.12.0, just some updates for shipping and testing to ensure conda-forge build succeeds (see #490).

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by jgliss over 3 years ago

pyaerocom - v0.12.0

pyaerocom version 0.12.0 (release summary)

This release comes with many new features, major improvements and a more stable API. Please see the individual points below for major changes.

The notes below summarise the major updates since the release of version 0.10.0. This includes the changes that were done in the recent release of version 0.11.0, which is the last version that officially supports the former "Aerocom Evaluation" tools, e.g. used extensively in CAMS61 project.

Major updates

NEW AeroVal tools

The web processing tools have seen major improvements and the backend has a new API, which can be found in the aeroval sub-package. Compared to the former AeroCom Evaluation tools (web sup-package in former releases), the new tools are a lot more flexible and support, for instance:

  • Processing of time series of statistical parameters.
  • Option to calculate seasonal statistics.
  • Temporal and spatial correlation in heatmap display.
  • Combination of different observation datasets into "superobservations".
  • All this is packed into a new modular and intuitive API.

Visit AeroVal: https://aeroval.met.no/
Follow AeroVal on twitter: https://twitter.com/AeroVal_MetNo

NOTE: web sub-package is deprecated

  • All code under pyaerocom/web is deprecated and this sub-package does not exist anymore in v0.12.0. Relevant code has been shipped and redesigned into the code basis for the new AeroVal tools (pyaerocom/aeroval subpackage).
  • Processing tools for old trends interface are deprecated. Instead trends visualisation is included in the new AeroVal interface and can be processed via the new tools in the aeroval subpackage.
  • Former CLIs for web tools (pyaeroeval, pyaerotrends) are deprecated. However, the simple pyaerocom (pya) CLI is still available.

New observation networks

  • Added support for reading of EEA air pollution data
  • Added support for reading of AirNow air pollution data
  • Added support for reading of MarcoPolo air pollution data

Further updates and improvements

  • Improvements in EBAS reader, e.g.
    • Evaluation of individual timestamps to resolve ts_type of NASA Ames file.
    • Improved logic for automatic column selection.
    • Automatic conversion of vmr to conc and vice versa (e.g. vmro3 to conco3)
  • Evaluation of wet deposition and precipitation data.
  • Revised, more powerful and intuitive co-location routines.
  • Improved and more robust unit conversions
    • E.g. Automatic detection and correction of implicit deposition units to
      explicit units, both for model and obs, centralised in units_helpers.
      py       module.
  • Improved robustness and performance of EMEP reader.
  • Support reading and co-location of AeroCom model climatology files (year
    9999 in filename). See here
    for details.
  • New features in TsType class (e.g.infer ts_type from total seconds)
  • Improved and more clearly defined API.
  • Installation with all requirements via PyPi
  • Many bug fixes
  • Improved test coverage.

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by jgliss over 3 years ago

pyaerocom - 1st release candidate for v0.12.0

This is a pre-release for v0.12.0, release summary and changelog will follow.

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by jgliss over 3 years ago

pyaerocom - v0.11.0

Las release to support the old aerocom evaluation tools!

Practice release for new release managers.

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by jgriesfeller over 3 years ago

pyaerocom - Version used for 2021 EMEP report (trends analysis)

NOTE: this is not an official release but a pre-release intended for reproducibility of the trends calculation done for the 2021 EMEP report. This corresponds to a development branch for release of pyaerocom version 0.12.0.

The code used to compute the trends for the EMEP report can be found here.

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by jgliss over 3 years ago

pyaerocom - Version 0.10.1: Minor updates and bug fixes

This is just a minor release that provides minor updates. That is, it includes

  • quick fixes for #319 and #322 ,
  • more restrictive with respect to dependencies, due to recent updates in pandas and iris, and
  • dependency netcdf4 was removed
  • the official documentation was moved from pyaerocom.met.no to ReadTheDocs (https://pyaerocom.readthedocs.io)

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by jgliss about 4 years ago

pyaerocom - Version 0.10.0

pyaerocom version 0.10.0

This release comes with many new features, major improvements and a more stable API. Please see the individual points below for major changes.

New feature modules

Modules that contain new features. See here for a list of all file modifications between version 0.8.0 and 0.10.0.

  • pyaerocom/combine_vardata_ungridded.py: Colocation of ungridded data.
  • pyaerocom/helpers_landsea_masks.py: Access and helpers for land-sea mask filtering.
  • pyaerocom/io/read_ghost.py: reading routine for GHOST dataset.
  • pyaerocom/io/read_mscw_ctm.py: Reading interface for EMEP data.
  • pyaerocom/molmasses.py: helpers related to access of molecular masses for species.
  • pyaerocom/trends_engine.py: Interface for computing trends using the method by Mortier et al., 2020.
  • pyaerocom/web/helpers_evaluation_iface.py: helper methods for conversion of ColocatedData to json files for Aerocom Evaluation interface.
  • pyaerocom/web/helpers_trends_iface.py: helper methods for conversion of ColocatedData to json files for Aerocom trends interface.
  • pyaerocom/web/utils.py: High-level methods based on results from standard Aerocom analysis. Currently, this contains a method compute_model_average_and_diversity which can be used to compute ensemble median or mean modeldata (e.g. used to compute AEROCOM-MEDIAN and MEAN in Gliss et al., 2020).
  • pyaerocom/scripts/cli.py: simple command line interface (currently very limited).
  • pyaerocom/scripts/highlevel_utils.py: high-level functions used in CLI.
  • pyaerocom/testdata_access.py: helpers related to initialization and access of pyaerocom testdata.

Updates related to supported observation data-sets and naming conventions

  • Support for reading GHOST data.
  • Support for reading of EMEP model data.

Below, relevant changes applied to already existing code are summarized.

Reading of data (io sub-package)

Reading of gridded data

  • ReadGridded class

    • Implement logic to apply constraint reading (i.e. read AOD where AE<0.5, read scattering where RH<40%, etc.). See input arg. constraints in ReadGridded.read_var and associated new class methods listed below.
    • Improve flexibility related to multiple vert_code matches using new method get_vert_code in Variable class.
    • Improve logic associated with resolving variable names for edge cases that include usage of alias variables and auxiliary reading of derived variables.
    • Add new default auxiliary / derived variables: sc550dryaer, concox, vmrox, fmf550aer.
    • New attrs. (incl. @property decorators): vars_provided, registered_var_patterns.
    • New methods: reinit, check_constraint_valid, apply_read_constraint, __repr__.

  • New: Reading interface and support for EMEP file conventions (class ReadMscvCtm).

Reading of ungridded data

  • ReadUngridded

    • Add option only_cached in read method, to only read cached data e.g. when working offline.
    • ReadUngridded can post compute variables with merge method combine.
    • New attrs. (incl. @property decorators): data_dir, post_compute
    • New methods: read_dataset_post, get_vars_supported

  • NEW: Reading interface for GHOST data (class ReadGhost).

  • read_ebas.oy (ReadEbas and ReadEbasOptions)

    • Introduced more reading options (e.g. related to shifting of wavelengths and assumed Angstrom Exponents used for shifting).
    • NOTE: Before, data at wavelengths (wvl) within a tolerance of +/- 50nm were considered around the output wvl (e.g. 500-600nm for sc550aer) and were used as is (i.e. without converting the values to 550nm). Now, the column with wvl closest to the output wvl is used and the measurement data is shifted to the output wvl (e.g. 550nm) assuming a typical Angstrom Exponent (AE=1.5 for extinction and scattering, AAE=1 for absorption, note also upcoming changes for next major release version 0.11.0).
    • Remove filelog in ReadEbas.
    • Resolve some issues related to old variable names and aliases.
    • Implement unit check in column selection method _find_best_data_column.
    • Add framework to output data.
    • Added attrs. in ReadEbas (incl. @property methods): sqlite_database_file, file_dir.

  • ReadUngriddedBase (template metaclass for ungridded reading)

    • allow to specify dataset_path (less dependency on server access, easier to work with local datasets)
    • New attrs. (incl. @property decorators): data_id
    • New methods (implemented): var_supported

Further updates in io sub-package

  • cachehandler_ungridded.py (CacheHandlerUngridded class)

    • Modified to handle custom filenames (e.g. used in new method UngriddedData.save_as).
    • New method delete_all_cache_files
    • New option force_use_outdated

  • Add method get_ungridded_reader in io/utils.py.

Data objects

  • GriddedData

    • Can be converted to xarray.
    • More robust and flexible time-series extraction, e.g., constraints can be applied during temporal resampling (i.e. hierarchical min_num_obs or how).
    • Improving automatic retrieval of lowest layer for profile data using CF attr. "positive".
    • Method mean now uses area weighted mean by default.
    • New methods: years_avail, split_years, mean_at_coords, filter_altitude, filter_region, apply_region_mask, aerocom_savename, to_xarray, area_weighted_mean

  • UngriddedData

    • Can be filtered by country names
    • Can now be saved as pickled objects
    • Create UngriddedData from StationData object(s)
    • Colocation of UngriddedData is now possible (relevant code is in new module combine_vardata_ungridded.py)
    • Support for wildcards in station data conversion methods.
    • New attrs. (incl. decorators): last_meta_idx, nonunique_station_names, countries_available
    • New methods: from_station_data (static method), check_set_country, check_convert_var_units, filter_altitude, filter_region, apply_region_mask, colocate_vardata, save_as, from_cache (static method).

  • ColocatedData

    • Implement method to compute regional time-series.
    • Support automatic assignments of countries for each site.
    • Support computation of area weighted statistics in calc_statistics.
    • New attributes: has_time_dim, has_latlon_dims, countries_available, country_codes_available, area_weights
    • New methods: get_country_codes, calc_area_weights, flatten_latlondim_station_name, stack, unstack, check_set_countries, filter_altitude, apply_country_filter, set_zeros_nan, apply_region_mask, filter_region, get_regional_timeseries

  • StationData

    • Computation of climatological time-series
    • Improved handling of metadata access (get_meta) and merging of metadata
    • resample_timeseries was renamed to resample_time (but old version still works.)
    • new method StationData.copy

Colocation of data

Low-level colocation routines (colocation.py)

  • Outliers in gridded/gridded colocation are now removed in original resolution.
  • Gridded/gridded colocation now re-grids to the lowest of both resolutions.
  • Add option resample_how, which can also be applied hierarchical, like min_num_obs (e.g. used to resample O3max).
  • resample_how option in high level colocation routines.
  • Option to use obs climatology for gridded / ungridded colocation.
  • New helper method correct_model_stp_coldata in colocation.py which applies STP correction to a colocated data object containing obs at STP based on station altitude and temperature derived from ERA Interim data (BETA feature only working for 2010 data and with access to METNo infrastructure).
  • Some bug fixes for certain edge cases.

High-level colocation routines (colocation_auto.py)

Affects classes ColocationSetup and Colocator

  • Support new EMEP reading routine.
  • Model and obsdata directories can be specified explicitly.
  • Option model_to_stp (BETA feature which will not work in most cases, see above).
  • New attrs. obs_add_meta, resample_how.
  • New methods (Colocator): read_model_data, read_ungridded

Filtering of data

  • Implement filtering of binary masks for GriddedData, UngriddedData and ColocatedData
  • Harmonize API of spatial filtering in data classes (i.e. method filter_region that can handle rectangular and binary region masks)
  • Automatic access to HTAP binary land-sea masks
  • Handling of binary and rectangular regions in Filter class.

Other updates (in top-level modules)

  • config.py (Config class)

    • Major improvements and API changes, e.g. related to automatic setup and adding new data search directories and ungridded observations.
    • In particular, attrs. BASEDIR, MODELBASEDIR, OBSBASEDIR are deprecated.
    • Instead, methods add_data_search_dir and add_ungridded_obs can be used to register data locations.

  • geodesy.py: new methods calc_latlon_dists, find_coord_indices_within_distance, get_country_info_coords.

  • New methods in helpers.py, the most relevant ones are extract_latlon_dataarray, make_dummy_cube_latlon, numpy_to_cube, sort_ts_types, calc_climatology.

  • New methods in mathutils.py: weighted_sum, sum, weighted_mean, weighted_cov, weighted_corr, corr (which were implemented in calc_statistics), vmrx_to_concx, concx_to_vmrx.

  • New class AerocomDataID in metastandards.py (is used in ReadGridded to separate data ID into metadata based on AeroCom 3 conventions).

  • New helper classes ObsVarCombi and AuxInfoUngridded in obs_io.py.

  • region.py: New method Region.plot and support for binary regions.

  • TimeResampler: implement handling of how to specify aggregating kernel (e.g. mean, max, min, std...).

  • New and interactive tutorials, for details see pyaerocom-tutorials repo.

  • NEW: colocation routine for ungridded data, the relevant code is in ne module test_combine_vardata_ungridded.py). NOTE: currently only implemented to create colocated UngriddedData objects, a routine that outputs ColocatedData object will come in v0.11.0.

  • tstype.py (class TsType)

    • Implement setter method for mulfac
    • New attrs (incl. @property decorators): TO_NUMPY, RS_OFFSETS (not in use), TSTR_TO_CF, datetime64_str, timedelta64_str, cf_base_unit, next_lower, next_higher
    • New methods: to_timedelta64, valid, to_numpy_freq
    • Rename method to_pandas to to_pandas_freq

  • variable.py

    • Define wildcard based lookup of default vertical codes for variable families (in VarNameInfo) and related class method get_default_vert_code (experimental)
    • new helper function get_emep_variables
    • Variable class
      • New attrs (incl. @property decorators): var_name_aerocom, default_vert_code, var_name_input, is_3d, is_wavelength_dependent, is_dry, is_alias
      • New methods: get_default_vert_code, __eq__
    • VarCollection class
      • support adding new variables (in addition to the ones defined in an ini file)
      • New methods: add_var, get_var

  • NEW toplevel method pyaerocom.initialise_testdata(): checks access to testdataset and downloads it to directory ~/MyPyaerocom if not available (or outdated).

Web processing tools (web sub-package)

  • Better separation of modules and code for AerocomEvalation tools and AerosolTrends tools.
  • Add support for auxiliary / derived obsdatasets in web.helpers.ObsConfigEval.
  • New method to compute ensemble median or mean in web/utils.py.

Aerocom Evaluation tools

  • Store a copy of the config file in experiment output directory.
  • Option weighted_stats in AerocomEvaluation (if active, weighted statistics are applied to gridded/gridded colocated objects in heatmaps).
  • Create json files for daily and monthly heat-maps.
  • Support also country based regional statistics (AerocomEvaluation.regions_how)
  • Regional time-series are now computed automatically for all regions.
  • Support evaluation of diurnal (hourly data).
  • utility function compute_model_average_and_diversity now also outputs fields for 1. and 3. quantiles (median) and std. (mean)
  • New options --onlyjson and --warnings in CLI pyaeroeval.
  • Improved robustness and flexibility of evaluation framework.
  • New methods and attrs. in AerocomEvaluation class:
    • Attrs: only_json, weighted_stats, regions_how, region_groups, resample_how, all_obs_vars, iface_names, name_config_file, name_config_file_json,
    • Methods: get_model_name, get_diurnal_only, read_model_data, read_ungridded_obsdata,

Trends evaluation tools

No relevant changes here (only some refacturing).

Plotting (plot sub-package)

  • New method plot_nmb_map_colocateddata in plot/mapping.py for plotting bias maps from ColocatedData
  • Improve options in heatmap plot (plot/heatmaps.py) (i.e. smart automatic formatting of heatmap annotation if values span several orders of magnitude).
  • Some bug fixes associated with automatic retrieval of colorbar levels (io/helpers.py).
  • New method init_multimap_grid in plot/mapping.py
  • Annotation fontsize in plot_scatter can now be controlled now via input arg fontsize_annot

Major API changes (not backwards compatible):

  • Renamed class ReadSulphurAasEtAl to ReadAasEtal
  • StationData.resample_timeseries is deprecated (but still works) and usage of new method resample_time is recommended
  • Some AeroCom variables were renamed (e.g. scatc550aer -> sc550aer, absc550aer -> ac550aer, bscatc* -> bsc*, etc.)

Testing and CI

  • Most tests now uses a publicly available test dataset.
  • Implemented automatic CI testing in Github Actions.
  • Add conftest.py for defining session wide test fixtures.
  • Many more tests resulting in largely improved test coverage, see here.

Bug fixes

  • Fix bug related to merging of StationData in case of overlapping time-series (issue 106).
  • Fix bug in method GriddedData.change_base_year
  • ReadEbas: Fixed bug related to column selection for wavelength range which identified data columns as valid if there was only one column match for var of interest.
  • Fix minor bug in reordering of dimensions of GriddedData when one dimension definition was missing.
  • Fix bug in GriddedData.crop due to time bounds not removed correctly.
  • Fix bug related related to extraction of time interval in GriddedData.sel (see here for details).

Upcoming changes

See here for a list of changes planned for the next upcoming release.

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by jgliss over 4 years ago

pyaerocom - Release version 0.8.0

Published here with one year delay, for completeness.

This release comprises major improvements, changes and many new features compared to the last release (comprising about 10 months of development time). Thus, below we only summarise the most important changes. For a list of all changes, please see the changelog file of this release (in subdirectory changelog).

  • New sub-package web (tools for high-level web processing)

    • Contains frameworks and routines for high level analysis of data and computation of json files both for AeroCom evaluation and trends web interfaces.
    • Includes 2 simple command line interfaces pyaeroeval and pyaerotrends for web processing
    • Main classes:

  • Gridded reading (ReadGridded class and methods used therein)

    • data_dir can be provided directly on input in ReadGridded (e.g. for working locally with no database access). However, data files are required to be in AeroCom naming convention.
    • og550gtaer is now primarily computed via od550aer-od550lt1aer
    • easy file filtering for all attributes accessible via filename (e.g. model, variable, year, vertical type)
    • option to compute variables during runtime for custom methods
    • Clean up of outdated methods
    • improved logic of processing work-flow, especially for computation of variables and handling of 4D files, e.g.
      • use ModelLevel files if Surface is requested but not available
      • Compute mass concentration fields (concXX) from mass mixing ratios (mmrXX) and density (rho) fields
    • Reading of climatological data (i.e. year 9999 in filename). Remark: tricky, because pandas cannot handle timestamps with year 9999
    • More flexible options for reading of iris cubes (iris_io.py)
    • Improved check and correction of invalid time dimensions in source files

  • Ungridded reading (Reading of observations)

    • EBAS: implemented framekwork for computation of variables from variables that can be read (or computed)
    • EBAS: evaluate and use flag columns (flagged data added to new flag column in UngriddedData object)
    • EBAS: support all occurring sampling frequencies (e.g. weekly, 2daily, etc.)
    • EBAS: default now reads raw (i.e. as is in NASA Ames files), but writes all relevant information for filtering (e.g. datalevel, flags) into output UngriddedData object, which can then be filtered flexibly after reading
    • New reading routine for GAW ascii format
    • New reading routine for data subset from trends paper by Aas et al.
    • Updated EARLINET reading routine after major changes in format (Feb. 2019)
    • More flexible handling of cached data objects in ReadUngridded (cf. changes in caching strategy below)

  • Data classes

    • StationData

      • Merging of multiple instances possible (including metadata merging and handling of overlapping data)
      • Removed attrs. stat_lon, stat_lat, stat_alt
      • Support trends computation and visualisation
      • Support profile data

    • UngriddedData

      • Support for flags and error data
      • Outlier removal
      • More flexible and robust conversion into StationData
      • More advanced filtering and subsetting (e.g. extract single variable)
      • Methods for merging of several instances
      • Added iter method (looping over data object -> returns StationData at each index, BETA)

    • GriddedData

      • More flexible subsetting (e.g. sel method)
      • Method to automatically infer surface level for 3D data
      • Cleaned up attributes: now everything is stored within underlying cube (i.e. attr. suppl_info is deperecated)
      • Added CF attributes such as standard_name and long_name
      • WORK IN PROGRESS: altitude access for 4D fields via get_altitude method (cleaned up and refactored old code due to below mentioned updates in mod vert_coords.py)
      • option to add metadata when converting to timeseries (StationData) at distinct locations

    • ColocatedData

      • Updated I/O and naming conventions
      • Region filtering
      • time resampling

    • All data classes contain many more helper and analysis methods and attributes, that are not explicitely mentioned here, for details see changelog

  • Colocation: Improved flexibility and robustness of colocation routines (modules colocation.py, colocation_auto.py), e.g.

    • more control on individual outlier removal for both input datasets
    • hierarchical resampling
    • option for outlier removal
    • option for unit harmonisation
    • option for colocating time before downsampling
    • option to ignore certain station names (for gridded / ungridded colocation)
    • colocation with climatology data
    • High level interface (Colocator class) for automatic colocation, e.g. used in AerocomEvaluation class for web processing.

  • Other changes:

    • Updated method calc_statistics: biases (NMB, MNMB) and FGE are now computed only from positive values
    • New modules units_helpers.py providing custom unit conversion, e.g. for non-CF conform units in data files (e.g. sulphur specific mass concentration data: ug S m-3)
    • Improved caching stragegy (now single variable instances of UngriddedData are cached)
    • Easier installation options
    • Support for simple geographical calculations
    • New helpers and processing methods in region.py
    • Support for more variables
    • Advanced and unified time resampling in TimeResampler class
    • More CF-compliant (e.g. units attr. in data classes)
    • More flexible and unified handling (and sharing) of metadata among different data objects
    • Methods for trends computation (class TrendsEngine)
    • Major improvements in ungridded caching using single variable cache files for I/O
    • Bug fixes
    • New class TsType for handling and comparing temporal resolutions (in new mod tstype.py)
    • More flexible tests (using pytest markers that check access to database)
    • Worked on implementation of vertical coordinate to altitude conversion methods (WORK IN PROGRESS, mod. vert_coords.py)

  • API changes:

    • Station class is deprecated
    • ReadGriddedMulti is deprecated (but still works)
    • sconc variables are deprecated (but still work): use conc instead (e.g. concso4 instead of sconcso4)
    • Renaming of classes / modules:
      • AllVariables to VarCollection
      • unit to units
      • Moved GridIO class from config.py to dedicated new module grid_io.py
      • Global setup dictionaries for time conversion moved from helpers.py to time_config.py

  • Not finished / under development / coming soon

    • Handling of vertical model coordinates
    • Colocation of profile data
    • Filtering by land / sea masks
    • Computation of regional average time series in data objects

  • Planned major changes for v0.9.0:

    • API refactor: StationData based on xarray.Dataset (currently variable data can be either numpy array, pandas Series or xarray)
    • Include filtering using land / sea masks (should work for GriddedData, UngriddedData, ColocatedData)
    • 4D data (ModelLevel):
      • conversion of vertical level coordinates to altitude
      • profile colocation (would add additional vertical dimension to Colocateddata)
      • Retrieval of aerosol layer height (PRODUCT)
      • Default vertical domains for vertical aggregation (particularly for web interfaces, e.g. 0-2km, 2-6km, >6km)

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by jgliss over 4 years ago

pyaerocom - Version used in Mortier et al., ACP, 2020

This is the pyaerocom version used for the processing in Mortier et al., 2020 paper

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by jgliss over 4 years ago

pyaerocom - Fixed some issues related to installation and import

Compared to the recent release v0.7.1, this release only comprises changes related to the library installation (both from source and using conda) and some improvements and bug fixes related to how pyaerocom sets up the output directories (e.g. cache directory) when imported.

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by jgliss over 6 years ago

pyaerocom - Version 0.7.0 with improved flexibility and user-friendliness

This release does not include many new features but rather significant improvements in the user-friendliness and flexibility.

Main improvements include (for details see changelog):

  • Improved setup of Config class (pyaerocom.const) for different path environments (including bug fixes)
    • leading to a more flexible and faster import of pyaerocom
  • Improved handling and flexibility of gridded data that is not prepared following the AeroCom standards
  • Much improved reading and processing of EBAS data (and ungridded data in general, e.g. EBAS SQL constraints can now also be provided in ReadUngridded.read())
  • Some new methods and bug fixes in UngriddedData class
  • pyaerocom can now be installed via conda (see README)

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by jgliss over 6 years ago

pyaerocom - Minor updates and bug fixes

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by jgliss over 6 years ago

pyaerocom - First release

First release of pyaerocom. For an overview of relevant features and usage, see introduction tutorial.

Remarks

  • Not yet ready to be used on Windows machines.
  • For data access, please see here or contact us.
  • Python 3 only

Atmosphere - Atmospheric Chemistry and Aerosol - Python
Published by jgliss over 6 years ago