A curated list of open technology projects to sustain a stable climate, energy supply, biodiversity and natural resources.

Recent Releases of harp

harp -

📅 2025-03-16

  • The rebin() operation now takes into account existing weight variables
    and properly handles angle variables. Rebinning the result of a
    bin_spatial() operation to a lower resolution now gives the same result
    as when a spatial bin was performed directly to that lower resolution.

  • Added harp_convert_unit_double() and harp_convert_unit_float().
    harp_convert_unit() is now deprecated.

  • Performing a unit conversion on single precision floating point
    variables will no longer convert the variable to double precision.

  • Updates to product ingestions:

    • GOME2 L2: added support for NHP/OHP ozone profile products and all
      products now use float instead of double for variables.
    • GEOMS FTIR: reinterpret vmr from old GEOMS templates as dry air vmr.
    • EarthCARE L1: add support for L1 ATLID, BBR, and MSI products.
    • EarthCARE L2: all vertical profiles are inverted to make them
      ascending, fixed geoid offset correction, and fixed AM__CTH_2B
      ingestion.
    • S5P PAL L2: added support for Ocean Colour (L2__KD____) product.

Sustainable Development - Environmental Satellites - C
Published by svniemeijer about 1 month ago

harp -

📅 2024-11-11

  • Fixed appending non-float variables with different sub-dimensions when
    using the Python harp.concatenate() function.

  • Updates to product ingestions:

    • ESACCI_GHG: fixed incorrect unit for CH4/CO variables
    • GOME-2 L2: added support for O3MARS aerosol layer height and added
      support for O3MARP V5
    • S5P L2: added snow_ice_type/sea_ice_fraction ingestion for AAI/CO/CH4
    • S5P L2 CH4: added 'corrected' option to ingest destriped columns and
      fixed issue in NIR/SWIR selection of surface albedo
    • S5P PAL L2: added support for SRON/SICOR HDO (L2__HDO__S) product

Sustainable Development - Environmental Satellites - C
Published by svniemeijer 6 months ago

harp -

📅 2024-07-30

  • Fixed numpy 2.x compatibility issues in python interface.

  • Added derivations to retrieve dry air mass/volume mixing ratio from H2O
    mass/volume mixing ratio with regard to dry air.

  • Fixed some issues with building the HARP R interface extension.

  • Updates to product ingestions:

    • S5P PAL L2 AOT: added support for cloud_fraction vs.
      effective_cloud_fraction split

Sustainable Development - Environmental Satellites - C
Published by svniemeijer 9 months ago

harp -

📅 2024-06-12

  • Updates to product ingestions:
    • EARLINET: fixed memory issue when reading extinction data.
    • EarthCARE: updates for new V11.2 format definition baseline
    • GEOMS UVVIS.DOBSON: fixed issue with reading of int16 typed data.
    • S5P PAL L2 SO2 COBRA: fixed cloud_fraction option behaviour and added
      alternative qa value calculation for processor V02.00.00.

Sustainable Development - Environmental Satellites - C
Published by svniemeijer 11 months ago

harp -

📅 2024-03-15

  • Fixed incorrect weight division for random uncertainties in spatial binning.

  • Added check for unit equality when appending variables.

  • Fixed CMake build for locating Python 3.12.

  • Updates to product ingestions:

    • EarthCARE: updates for new V11 format definition baseline
    • ECMWF GRIB: added general support for new masterTablesVersion numbers
    • S5P L2 CH4: added support for nir aerosol/cloud/surface property ingestion.
    • S5P L2 CLOUD: added support for nir-band spatial grid data
    • S5P PAL L2: added support for OClO
    • S5P PAL L2 AOT: added support for V02.00.00 output format
    • S5P PAL L2 BRO: fixed memory issue with reading total column and
      uncertainty variables
    • S5P PAL L2 TCWV: fixed wrong quantity for density variables

Sustainable Development - Environmental Satellites - C
Published by svniemeijer about 1 year ago

harp -

📅 2023-11-13

  • Fill values of floating point types are now explicitly set to NaN when
    exporting to a netcdf or hdf5 file.

  • Updates to product ingestions:

    • EARLINET: fixed issue where no backscatter/extinction data was ingested.

Sustainable Development - Environmental Satellites - C
Published by svniemeijer over 1 year ago

harp -

📅 2023-10-12

  • Updates to product ingestions:
    • IASI L1C: added missing angle variables
    • S5P PAL L2 AER_OT: added missing wavelength variable
    • S5P PAL L2 CHOCHO: fixed missing units for number density variables

Sustainable Development - Environmental Satellites - C
Published by svniemeijer over 1 year ago

harp -

📅 2023-09-19

  • Updates to product ingestions:

    • GEOMS FTIR: added support for C2H3NO5/PAN and C2H4, added support for
      template V3.
    • GEOMS TCCON: added support for template V6.
    • S5P L2 ALH: add support for revised surface_albedo variable and added
      surface_albedo_uncertainty.
    • S5P L2 O3: fix custom V1 QA filter for NRTI products
    • S5P PAL L2: fixed memory issue when reading aerosol_optical_depth_validity
      of AER_OT product.
  • Fixed missing documentation on weight factor handling in binning algorithm.

  • Updated embedded expat library to version 2.5.0.

  • Fixed issue where jpeg dll was missing from windows binary installer.

Sustainable Development - Environmental Satellites - C
Published by svniemeijer over 1 year ago

harp -

📅 2023-07-24

  • The python harp.import_product() function now allows HARP .pth files to be
    provided as input.

  • Reintroduced propagation of *_uncertainty_random variables when performing
    spatial binning with area weighted averaging.

  • Allow regridding using a multi-dimensional source axis-variable as input.
    e.g. convert from an 'altitude {latitude,longitude,vertical}' grid to
    a fixed 'altitude {vertical}` grid.

  • Added documentation for regridding/binning algorithms.

  • Updates to product ingestions:

    • ACE-FTS: fix ingestion of ASCII formatted products
    • GOME-2 L2: added support for offline absorbing aerosol index (O3MARP)
    • EARLINET: added support for new format (removed support for old format)
    • EarthCARE: added support for AM__ACD_2B, AM__CTH_2B, BM__RAD_2B,
      BMA_FLX_2B, MSI_AOT_2A, MSI_CM__2A, and MSI_COP_2A
    • ECMWF GRIB: added support for new masterTablesVersion=30 data
    • S5P PAL L2: added support for AER_OT, BRO, CHOCHO, SIF, SO2 COBRA, and
      TCWV

Sustainable Development - Environmental Satellites - C
Published by svniemeijer almost 2 years ago

harp -

📅 2023-06-08

  • Rename all uses of 'water vapour' to 'water vapor'.

  • Added =| bitfield operater to allow filtering for the case where only some
    bits within the mask are set.

  • The uncertainty propagation of *_uncertainty_random variables for the
    temporal binning operation now always assumes no correlation. For spatial
    binning the random component is removed for the area weighted average
    approach and it is propagated using no correlation for the simple binning
    based on center points.

  • Added derivation for 'area {latitude,longitude}' and fixed area calculation
    where South Pole was an edge point.

  • Fixed reading of string arrays in HARP Python interface.

  • Updates to product ingestions:

    • ECMWF GRIB: added support for new masterTablesVersion=29 data
    • S5P L2 CLOUD: fixed qa_value ingestion for CRB (now use qa_value_crb)
    • S5P L2 SO2: added support for SO2 layer height
    • OMI L3: added support for OMHCHOd, OMNO2d, and OMSO2e

Sustainable Development - Environmental Satellites - C
Published by svniemeijer almost 2 years ago

harp -

📅 2022-12-12

  • Added additional conventions for variable names related to areosol/cloud
    properties.

  • Fixed issue with geographical area overlap calculation and improvided
    performance of performing operations on geographical polygon data.

  • Windows .msi installer now allows HARP component selection for core, Python,
    IDL, MATLAB, and R.

  • Updates to product ingestions:

    • AEOLUS L2B: properly split datetime, datetime_start, and datetime_stop
    • CCI L3 O3 LP: fixed wrong variable name
    • EarthCARE: added additional product types and updated ACM_CAP_2B
    • ECMWF GRIB: changed RWC/SWC/LWC/IWC variable names
    • OSIRIS L2: fixed wrong variable names
    • TES TL2ATMTN/TL2ATMTL: fixed wrong variable names

Sustainable Development - Environmental Satellites - C
Published by svniemeijer over 2 years ago

harp -

📅 2022-10-31

  • Improved algorithm to calculate center point from latitude/longitude bounds
    polygon.

  • Add derivation of time dependent from time indepent for datetime variables.

  • Updates to product ingestions:

    • GEOMS PANDORA: added systematic/random uncertainty for effective
      temperature, renamed H2CO in harp variable names to HCHO.
    • OCO: added support for ACOS/OCO-2/OCO-3 L2/L3 products.
    • Ozone CCI L3 LP: fixed memory issue when reading
      merged_ozone_concentration.
    • S5P L2 HCHO: added amf=clear_sky ingestion option

Sustainable Development - Environmental Satellites - C
Published by svniemeijer over 2 years ago

harp -

📅 2022-05-18

  • Updates to product ingestions:
    • CLOUDNET L2 classification: fixed fillvalue mapping of values to NaN.
    • GEOMS PANDORA: fixes for template V003 support
    • GEOMS FTIR: allow instrument lat/lon/alt variables to be one-dimensional
      with replicated values.
    • GEOMS UVVIS.BREWER: added random+systematic uncertainty variables.
    • S5P L2 CO: fixed vertical ordering of new apriori profile.

Sustainable Development - Environmental Satellites - C
Published by svniemeijer almost 3 years ago

harp -

📅 2022-03-14

  • Fixed issue where bin() operation was removing scalar enumeration variables.

  • Updates to product ingestions:

    • ECMWF GRIB: fixed parameter mapping for CAMS model data that uses
      masterTablesVersion=27 data and added support for new
      masterTablesVersion=28 data
    • GEOMS UVVIS.BREWER: added support for template V001
    • GEOMS PANDORA: added support for template V003
    • S5P L2 AAI: added new 335/367 wavelength pair
    • S5P L2 CO: added new apriori profile and support for new AVK unit
    • S5P L2 O3 Profile: added surface_albedo and cloud_albedo
    • S5P L2 HCHO/O3/SO2: added support for updated NRTI indication attribute.

Sustainable Development - Environmental Satellites - C
Published by svniemeijer about 3 years ago

harp - 1.14

📅 2021-10-20

  • Updated embedded udunits2 library to version 2.2.28.

  • Improved CMake build system.

  • Fixed issue when building against HDF5 1.12.

  • Updates to product ingestions:

    • AEOLUS L1B: fixed ingestion of Mie/measurement data; added sensor angles
    • AEOLUS L2B: fixed ingestion of Mie data; added sensor angles
    • CLOUDNET L2 classification: added support for new (netcdf4) format
    • EARTHCARE: added very early support (limited version of ECA_ACM_CAP_2B)
    • ECMWF GRIB: fixed unit of backscatter coefficient 1/(milli sr) -> 1/(m sr),
      added support for new masterTablesVersion=27 data
    • GEOMS UVVIS.DOAS: allow template V007 files to be stored in hdf5 format
    • GHG CCI: added support for TROPOMI WFMD L2
    • OMI L2: added sensor alt/lat/lon and surface altitude/pressure
    • OMI L2 OMAERUV: fix ingestion issues
    • QA4ECV HCHO: added amf=clear_sky ingestion option
    • S5P L2 HCHO: renamed cloud_altitude variable to cloud_height
    • S5P L2 NO2: added retrieval of O2-O2 cloud (O22CLD) data
    • S5P L2 NO2+FRESCO: added scene_albedo and scene_pressure
    • S5P L2 O3 Profile: fixed wrong concentration variables (vmr -> nd)
    • S5P L2 SO2: updated SO2_type enumeration list

Sustainable Development - Environmental Satellites - C
Published by svniemeijer over 3 years ago

harp -

📅 2021-03-12

  • Added rebin() operation.

  • Added harp_product_rebin_with_axis_bounds_variable() function.

  • Added support for datetime_bounds variable.

  • Fixed issue in =& operation (it returned true if any of the bits were 1
    instead of all bits).

  • Updates to product ingestions:

    • GEOMS: renamed all site_name variables to location_name
    • GEOMS FTIR H2CO: fixed issue in determining unit for H2O data (that would
      fail the ingestion)
    • S5P L2 CO: added co=corrected ingestion option
    • SCIAMACHY L2: fixed averaging of cloud fraction for co-added pixels

Sustainable Development - Environmental Satellites - C
Published by svniemeijer about 4 years ago

harp -

📅 2020-10-29

  • The csv output of 'harpdump --dataset' can now be used as content of a .pth
    file. This improves the performance of harpcollocate/harpmerge/etc. as they
    no longer need to open and extract metadata from files themselves anymore.

  • harpcollocate now stores differences as signed values in collocation result
    file (instead of absolute differences).

  • Added harp_dataset_prefilter(). This function is now also called within
    harpmerge and harpcollocate to pre-filter the input dataset(s) for datetime
    and collocate left/right filters that are at the start of the operations
    string.

  • Added format and history fields to harp_product_metadata.

  • Added harp.import_product_metadata() to Python interface.

  • harp_dataset_add_product() now ignores files/directories starting with '.'.

  • Added uncertainty propagation to bin() and bin_spatial() operations
    (using uncorrelated assumption for bin() and correlated for bin_spatial()).

  • Added global option to determine whether uncertainty propagation should
    use a correlated or uncorrelated approach. Currently only used for the bin()
    operation.

  • Added global option to create a collocation_datetime variable when performing
    a collocate_left() or collocate_right() operation.

  • Added wildcard pattern matching ('*'/'?') to keep() and exclude() operations.

  • Added derivation to retrieve angstrom exponent from aerosol optical depth.

  • Added derivation to retrieve tropospheric/stratospheric column avk from
    total column avk and tropopause altitude.

  • Fixed potential memory issues and improved overall code stability.

  • Updates to product ingestions:

    • S5P L2 CLOUD: added cloud_type and cloud_top_temperature
    • S5P L2 O3: added qa_filter=custom ingestion option to set validity based
      on official PRF recommendation for processor V1.x generated products
    • MLS L2: fixed memory issue in reading of validity variable and updated
      validity flag mapping to version 4.x-4.0 of the Data Quality document

Sustainable Development - Environmental Satellites - C
Published by svniemeijer over 4 years ago

harp -

📅 2020-05-19

  • Added derivations to retrieve column number density from column vmr.

  • Fixed issue in bin_spatial() where the weightfactor could be too high when
    oversampling the input data. Grid cells that intersected with the data
    polygon could end up with a weightfactor >1.0 (but <2.0).

  • Fixed issue where molar mass was incorrectly provided for the species with
    the shortest matching name. e.g. the molar mass of CO was returned for CO2
    and of NO for NO2. This impacts vmr <-> mmr conversions for BrO2, COF2, COS,
    CO2, C2H2O2, H2O2, NOCl, NO2, N2O, N2O5, and all isotopologues (e.g. H2O_161).

  • Removed irrelevant harp_str64() and harp_str64u() functions (HARP does not
    support 64bit integers).

  • Fixed issue where molar mass linked to variables with a 'surface',
    'tropospheric', or 'stratospheric' prefix returned 0.

  • Fixed error/crash in deriving dry air column vmr from dry air column mmr.

  • Updates to product ingestions:

    • GOME-2 L2: added support for SO2 columns at all plume heights
    • OMI L2 CLDO2/CLDRR: added validity variable

Sustainable Development - Environmental Satellites - C
Published by svniemeijer almost 5 years ago

harp -

📅 2020-03-31

  • Added support for -ar/--reduce-operations parameter in harpmerge and
    for reduce_operations parameter in harp.import_product() Python function.

  • Fixed incorrect "invalid variable" error on weight variables for angular
    quantities when performing a harpmerge.

  • Fixed issue where the udunits2 library could not be initialized if the
    environment had a LC_NUMERIC locale that did not have '.' as decimal
    separator.

  • Fixed issue in name mangling of embedded expat library.

  • Updates to product ingestions:

    • GOMOS L1 Limb: fixed unit of wavelength_photon_radiance (and uncertainty)
      and documented conversion algorithm
    • S5P L2 O3 PR: support for IODD 12.0.0
    • S5P L2 O3 TPR: removed support (future availability of product uncertain)
    • GEOMS PANDORA: allow for trailing \0 in global attribute values

Sustainable Development - Environmental Satellites - C
Published by svniemeijer about 5 years ago

harp -

📅 2019-12-01

  • Fixed issue where using the 'bin()' operation did not correctly create a
    '<variable>_weight' variable from a 'weight' variable when it was missing.

  • Added default units to documention on variable name conventions.

  • Updates to product ingestions:

    • Calipso L2: fixed unit of backscatter coefficient
    • ERS GOME L2: fixed unit of cloud top height
    • S5P L2 AER_LH: fixed unit of aerosol optical depth
    • S5P L2 SO2: add scaled avk for 1km/7km/15km box profiles

Sustainable Development - Environmental Satellites - C
Published by svniemeijer over 5 years ago

harp -

📅 2019-11-01

  • Fixed issue where results from deriving partial column densities from vmr and
    pressure were a factor 10 too low.

  • Fixed issue where operation following an index filter operation did not
    get executed.

Sustainable Development - Environmental Satellites - C
Published by svniemeijer over 5 years ago

harp -

📅 2019-10-21

  • Added operations to filter on dimension indices (e.g. 'index(vertical) < 5').

  • Changed bin() and bin_spatial() operations to use/produce separate 'count'
    and 'weight' variables. The counts/weights are used to allow the following
    commands to now be equivalent:
    harpmerge -a 'bin()' -ap 'bin()' a.nc b.nc merged.nc
    harpmerge -ap 'bin()' a.nc b.nc merged.nc
    As well as the following:
    harpmerge -a 'bin_spatial(...)' -ap 'squash(time, (latitude_bounds, longitude_bounds));bin()' a.nc b.nc merged.nc
    harpmerge -ap 'bin_spatial(...)' a.nc b.nc merged.nc

  • Fixed issue with invalid bounding box optimization for spherical area
    filter operations.

  • Fixed missing error handling when using invalid operations in
    harp.export_product() in Python.

  • Fixed Python3 compatibility issue when exporting string data.

  • Updates to ECMWF GRIB, IASI L1/L2, GOME-2 L1/L2, MIPAS L1, and S5P L2
    product ingestions.

Sustainable Development - Environmental Satellites - C
Published by svniemeijer over 5 years ago

harp - 1.8

📅 2019-07-31

  • Added R interface.

  • Improved derivation for tropopause altitude/pressure (to not return NaN).

  • Added derivations for gravity.

  • Added derivations to convert between total air and dry air mixing ratios.

  • Fixed valid() operation to properly use valid_min/valid_max attribute values.

  • Fixed issue in adding *_count variable in append/merge when count variable
    had more than one dimension.

  • Updates to ECMWF GRIB, GEOMS MWR, and S5P L2 product ingestions.

Sustainable Development - Environmental Satellites - C
Published by svniemeijer over 5 years ago

harp -

📅 2019-06-06

  • Added --no-history option to harpconvert, hardump, and harpmerge.

  • Added harp.execute_operations() and harp.convert_unit() to Python interface.

  • Added optional min_collocation_index/max_collocation_index arguments to
    collocate_left()/collocate_right() operations.

  • harp_product_set_history() and harp_product_set_source_product() now accept
    NULL as argument to clear the current value.

  • Appending/merging products now automatically adds missing *_count variables.

  • Fixed potential crash during initialization of HARP ingestion module.

  • Fixed invalid results when using bin_spatial with latitude_bounds {time,2}
    and longitude_bounds {time,2} variables (i.e. range instead of polygon).

  • Fixed array ordering in MATLAB interface. Multi-dimensional arrays are now
    represented from fastest running to slowest (=fortran ordering) as intended.

  • Update to S5P L2 product ingestion.

Sustainable Development - Environmental Satellites - C
Published by svniemeijer almost 6 years ago

harp -

📅 2019-04-29

  • Added clamp() and squash() operations.

  • Allow bin() and sort() operations to be performed using a set of variables.

  • harp_product_sort() now takes a list of variables.

  • The rename() operation no longer fails if the source variable does not exist
    but a variable with the target name already exists.

  • The python harp.import_product() function now uses the C library to perform
    the contactenation of products.

  • Added post_operations argument to python harp.import_product() function.

  • Besides using a polygon of >=3 vertices, a geospatial area can now also be
    defined via a bounding rect. Use e.g. 'latitude_bounds {time,2}' and
    'longitude_bounds {time,2}' to define the min/max of the bounding rect.

  • Added derivation for tropospheric/stratospheric column from partial column
    profile and tropopause.

  • Corrected derivation for calculating tropopause altitude/pressure.

  • Deriving a variable will now use the conversion that requires the minimum
    amount of intermediate variables to be calculated.

  • Added support for scene albedo/pressure/type variables.

  • Fixed issue where regridding in the vertical dimension using pressure as axis
    could end up with wrong pressure_bounds values.

  • Fixed problem in matlab mex extension determination on linux when using the
    configure script.

  • Fixed setting of default CODA definition path and udunits2 xml path for
    Python on Windows.

  • Added format, variable count, and dimension sizes to harpcheck output.

  • Added support for TCCON products in GEOMS format.

  • Updates to AEOLUS L2A, ESACCI_OZONE_L2_TC, GEOMS FTIR, GOME2 L2, OMI L2 NO2,
    QA4ECV, and S5P L2 product ingestions.

  • Updated embedded expat library to version 2.2.6.

Sustainable Development - Environmental Satellites - C
Published by svniemeijer almost 6 years ago

harp - 1.5

📅 2018-12-17

  • Fixed issue where vertical smoothing/regridding provided invalid results if
    input variables were available as float instead of double data.

  • harp_product_get_derived_variable() no longer converts data to double if
    no data type was provided and data already had the right unit.

  • Fixed issue where listing derivations for a specific variable would not show
    all possible conversions.

  • Add derivation for cloud height from cloud base/top height.

  • Added support for Dobson, Pandora and Sonde products in GEOMS format.

  • Updates to ACE-FTS, GOME/GOME-2 L2, and S5P L2 product ingestions.

Sustainable Development - Environmental Satellites - C
Published by svniemeijer over 6 years ago

harp -

📅 2018-09-28

  • Added -t/--target option to harpdump --list-derivations.

  • Added variable_name parameter to harp_doc_list_conversions().

  • Added support for meridional/zonal wind speed, and planetary boundary layer
    height variables.

  • Added support for sic (Shannon information content) variables for vertical
    profiles.

  • Aerosol optical depth and extinction coefficient variables will now
    automatically use a log/log regridding in the spectral dimension.
    This is equivalent to using the Angstrom Exponent approach.

  • Fixed interpolation issue where points beyond the source grid could end up
    having an arbitrary wrong value (instead of NaN). This is a regression of
    harp 1.3.

  • Add derivations between wind speed/direction and zonal/meridional velocity.

  • Flattening a dimension now special cases a dimension length 1 (with less
    modifications to variables/product).

  • Fixed issue where a point distance of 0 could sometimes result in NaN.

  • Updates to S5P L2, GOME/GOME-2 L2, GEOMS UVVIS-DOAS-OFFAXIS, ECMWF GRIB, and
    AERONET product ingestions.

Sustainable Development - Environmental Satellites - C
Published by svniemeijer over 6 years ago

harp -

📅 2018-07-05

  • Added bin_spatial() operation.

  • Improved binning operation to automatically filter for NaN values and add
    variable-specific count variables to specify the number of non-NaN values.

  • Performing a binning operation on top of a binning operation will propagate
    averages based on the available count variables (i.e. as if the final bin was
    performed on the original data).

  • Entries added to the global 'history' attribute now also include the current
    utc time and harp version before the execution command.

  • Allow NULL as argument for harp_variable_set_description() and
    harp_variable_set_unit() to clear the current description/unit.

  • Added harp_product_bin_spatial() and harp_variable_copy_attributes()
    functions.

  • Added optional parameter to regrid() operation to specify the axis boundaries.

  • Python harp.import_product() now ignores empty products when ingesting a list
    of products.

  • Added derivations:

    • column vmr/mmr from column density ratios
    • total column mass density from surface pressure and pressure profile
    • mass mixing ratio from mass density
    • between dry air and total air mass (column) density
    • between dry air, total air, and H2O number densities
  • Fixed derivation of altitude from gph/latitude.

  • Updates to S5P L2 and MIPAS L2 product ingestions.

Sustainable Development - Environmental Satellites - C
Published by svniemeijer almost 7 years ago

harp -

📅 2018-04-09

  • Allow data compression for HDF5 output:

    • Added harp_set_option_hdf5_compression() and
      harp_get_option_hdf5_compression() functions to C library.
    • Added --hdf5-compression option to harpconvert and harpmerge.
    • Added hdf5_compression parameter to harp.export_product() Python function.
  • Allow regrid() operation to interpolate in the time dimension.

  • Improved binning of angle values (average of 1 and 359 [deg] is now 0).

  • Added bin() operation variant without parameters.

  • The bin() operation using a collocation result file now retains the
    collocation_index variable (with the first value for each bin).

  • Changed smooth() operation such that each vertical level where the source
    grid point was NaN the target grid point also remains NaN.

  • Added -ap option to harpmerge.

  • Added --dataset option to harpdump and changed output format of
    harp_dataset_print() and harp_product_metadata_print() to csv format.

  • Added 'operations' parameter to Python harp.export_product() function.

  • Importing/exporting a product with operations in Python now automatically
    updates the history attribute.

  • Fixed issue where harpmerge did not introduce a time dimension for all
    variables.

  • Fixed issue where datetime_start and datetime_stop global attributes were not
    created when exporting in HDF4 or HDF5 format.

  • String attributes in HDF5 files can now be empty strings (using H5S_NULL).

  • Allow HARP to import HDF5 files that are generated by the netCDF4 library
    (allow numeric attributes to be 1-element arrays instead of scalars).

  • harpcheck now returns with exit code 1 if any of the imports fail.

  • harpcollocate --update now only allows one dataset as argument.

  • Fixed performance issue with spatial filtering on systems with slow math lib.

  • Fixed mapping description of GOME2 L1 product ingestion.

  • Fixed wrong results of area overlap calculation.

  • Replaced '_' by '-' in long versions of argument names of harpcollocate.

  • Added support for dfs (degrees of freedom for signal) profiles.

  • Updates to S5P L2 product ingestions.

Sustainable Development - Environmental Satellites - C
Published by svniemeijer about 7 years ago

harp - 1.1

📅 2017-11-06

  • Parameters that required a file or a directory as dataset reference can now
    also take a file with a .pth extension as input. This file should contain a
    line-separated list of paths to files and/or directories of the dataset.

  • Specifying a directory for a dataset will now recursively add all products
    (including sub-directories and contents of .pth files).

  • The variable that is binned on with a 'bin(variable)' operation is now
    always kept in the resulting product.

  • The bin() operation can now handle 'count' variables (summing it if present
    and creating it otherwise).

  • Fixed issue where collocate filter operation on a product with a single
    sample could end in an infinite loop.

  • Fixed issue in harpcollocate when providing a -nx option without a -ny.

  • Fixed issue with wrap() operation for values < specified minimum.

  • Fixed issue with collocate_left/collocate_right operations not including
    last sample when applying it on products in an external format.

  • harpcollocate now ignores empty products instead of raising errors for them.

  • Added -oa/-ob and -aa/-ab options to harpcollocate matchup.

  • Added solar_declination_angle and solar_hour_angle variables and derivations.

  • Added orbit_index variable and added this to various product ingestions.

  • Fixed derivations of solar azimuth/zenith angles from datetime and lat/lon.

  • Added derivations for averaging kernel conversions.

  • Added derivations for tropopause altitude/pressure.

  • Added NH3 (amonia) as supported species.

  • Updates to ECMWF-GRIB, GEOMS-TE-UVVIS-DOAS-OFFAXIS, GEOMS-TE-FTIR, and
    S5P L2 product ingestions (and several others for the new orbit_index).

Sustainable Development - Environmental Satellites - C
Published by svniemeijer over 7 years ago

harp - 1.0

📅 2017-09-25

  • The import and ingest functions have been combined into a single import
    function that can both read HARP compliant products as well as ingest from
    the supported external formats. This means:

    • harp_import() and harp_ingest() are combined into a single harp_import()
    • harp_ingest_test() has been renamed to harp_import_test()
    • harpfilter has been removed (use harpconvert instead)
    • '--test' option of harpconvert has been removed (use harpcheck instead)
  • harpcollocate has been completely rewritten.

    • any 1D time-dependent variable can now be a collocation criterium
    • nearest neighbour filtering with weighted average is no longer supported
    • collocation result files are not backward compatible due to a change in
      the CSV header
    • arguments to harp_collocation_result_new() and
      harp_collocation_result_add_pair() have changed
  • Added support for enumeration variables. Enumeration variables have a int8,
    int16, or int32 type and provide an index into a fixed list of enum names.
    Enumeration variables can be treated as string values in operations.

  • Added MATLAB and IDL interfaces for HARP.

  • Added -o option to harpdump and harpmerge.

  • Added -a and -o options for 'harpdump --list-derivations'.

  • '--list-derivations' option of harpconvert has been removed (use harpdump
    instead).

  • Added -l option to harpmerge.

  • Renamed area/point operations:

    • area_mask_covers_area -> area_covers_area
    • area_mask_covers_point -> point_in_area
    • area_mask_intersects_area -> area_intersects_area
    • point_in_area -> area_covers_point
      The first part of the operation name now reflects the item in the
      product and the second part reflects the parameter that is provided.
  • Improved performance of point/area vs. area overlap operations by first
    comparing against the bounding box of a polygon.

  • Added sort() and area_inside_area() operations.

  • The derive() operation can now be used without dimension specification
    to perform a unit and/or data type conversion on an existing variable.

  • The derive_smoothed_column(), regrid() and smooth() operations can now
    use a single merged collocated product (instead of a combination of a csv
    collocation result file and a dataset directory of collocated products).

  • An area parameter for operations can now either be a reference to an
    area mask file or a single polygon provided as lat/lon array parameters.

  • Added derivation to derive the area (size) of latitude/longitude bounds.

  • Added derivation to derive degree of freedom for signal (DFS) from 2D
    averaging kernels (AVK).

  • Derivations of bound variables will now clamp to the edge midpoints when
    the 'regrid_out_of_bounds' option is set to 'edge'.

  • The 'data_type' parameter for harp_product_get_derived_variable() and
    harp_product_add_derived_variable() is now a pointer to allow it to be
    optional (i.e. pass NULL to use the default data type).

  • Added harp_geometry_get_area(),
    harp_product_get_smoothed_column_using_collocated_product(),
    harp_product_smooth_vertical_with_collocated_product(),
    harp_product_sort(), harp_product_regrid_with_collocated_product(),
    harp_set_udunits2_xml_path(), harp_set_udunits2_xml_path_conditional(),
    and harp_variable_set_enumeration_values() functions.

  • harp_import_product_metadata() and harp_dataset_import() now have an
    'options' parameter.

  • Changed 'percentage' parameter of harp_geometry_has_area_overlap() to
    'fraction'.

  • harp_product_set_source_product() now has a product path as parameter
    and stores the base name of this as source_product value.

  • Added HARP_UNIT_AREA default unit.

  • The C interface for collocation results was modified to support a
    dynamic set of collocation criteria differences (based on variable names
    for these differences).

  • The Python interface now sets the udunits2 xml path dynamically,
    relative to the location of the HARP C library.

  • wrap() operation now properly updates valid_min/valid_max properties.

  • codadef files are now installed in /share/coda/definitions.
    This allows a present CODA installation to use the same codadef files.

  • Windows installation now installs data and documentation files in the
    subdirectory /share/harp.

  • Fix issue in deriving total column from partial column profile if partial
    column profile did not have {time,vertical} dimensions.

  • Fix invalid results from binning operation/function.

  • Improve speed of --list-derivations on a product.

  • Split data format documentation into seperate sections under a new
    heading 'conventions'.

  • Added product ingestions for Cloudnet L2a classification products.

  • Updates to ECMF-GRIB, GEOMS-TE-UVVIS-DOAS-OFFAXIS, MIPAS, QA4ECV, and
    S5P product ingestions.

Sustainable Development - Environmental Satellites - C
Published by svniemeijer over 7 years ago

harp - 0.6

📅 2017-04-23

  • Added ingestion support for ACE-FTS, CALIPSO, Cloud CCI, and NPP-Suomi.

  • Added bin() and set() operations.

  • Added optional data_type parameter to derive() operation.

  • Major refactoring of internal implementation of operations (greatly
    reducing and simplifying the code).

  • Fix application of apriori for smoothed column derivation (for vertical
    range where partial columns are NaN, don't subtract apriori*AVK, but just
    set to apriori).

  • Fix memory issue in partial column from vmr dry air derivation.

  • Remove double definition of radiance conversions.

  • Added global HARP option regrid_out_of_bounds.

  • Added harp_product_bin() function.

  • Add data_type parameter to harp_product_add_derived_variable() and
    harp_product_get_derived_variable() functions.

  • Added data_type parameter to harp_product_get_derived_variable() and
    harp_product_add_derived_variable() functions.

  • Allow 'Conventions' global attribute to contain more than just a reference
    to HARP.

  • Renamed O3MSAF codadef to ACSAF. When updating HARP, you should remove
    any existing O3SMAF-*.codadef files from your installation directory.
    Otherwise you will get 'detection rule for ... is shadowed by ... ' errors.

  • Fixed issue with TES TL2CH4L ingestion.

  • Updates to Aeolus, S5P, QA4ECV, and Ozone CCI product ingestions.

Sustainable Development - Environmental Satellites - C
Published by svniemeijer about 8 years ago

harp -

📅 2017-03-26

  • Changed Open Source licence of HARP from GPLv2 to BSD (3-Clause).
  • Changed all '-' to '_' in operation function names to make them identifiers.
  • Added harpmerge command line tool.
  • Added regrid(), rename(), smooth(), derive_smoothed_column(), point_in_area(), and wrap() operations.
  • Removed the harpprofile command line tool. Its functionality is replaced by the regrid() and smooth() operations.
  • Renamed harp_product_regrid_vertical_with_collocated_dataset() to harp_product_regrid_with_collocated_dataset() and harp_product_smooth_vertical() to harp_product_smooth_vertical_with_collocated_dataset()
  • Added vertical_unit parameter to harp_product_smooth_vertical() and harp_product_regrid_with_collocated_dataset() functions.
  • Replaced harp_product_regrid_vertical_with_axis_variable() with harp_product_regrid_with_axis_variable(). It can now be used for other dimensions too (except the time dimension).
  • Added harp_collocation_result_filter_for_collocation_indices(), harp_product_get_smoothed_column(),
    harp_product_get_smoothed_column_using_collocated_dataset(), harp_product_remove_variable_by_name(), harp_variable_append(), harp_product_append(), and harp_variable_smooth_vertical() functions.
  • Renamed harp_product_get_variable_id_by_name() to harp_product_get_variable_index_by_name().
  • Python harp.import_product() and harp.ingest_product() functions now accept a list of filenames or a globbing pattern as parameter (which will return the result of harp.concatenate() on the imported/ingested products).
  • Automatically set CODA definition path for HARP Python interface (path can be overruled with CODA_DEFINITION environment variable).
  • The parsing of the operations string has been rewritten using flex/yacc. This has fixed several issues related to the parsing and execution of operations.
  • Fixed issue with flatten() operation providing wrong results.
  • Fixed issue where vertical regridding to a common grid provided wrong results.
  • Fixed issue where interval interpolation for a pressure grid was not using a log-linear interpolation.
  • Aligned order of latitude/longitude arguments in all functions to have latitude first and longitude second.
  • Added unit 'ppv' to the included udunits2-common.xml file.
  • Added variables for surface temperature, total water vapour content, and total cloud cover to ECMWF GRIB ingestion.
  • The default internal units have been changes to SI (except for molar mass). All algorithms have been adapted accordingly.
  • Added variable conversions for: apriori profiles, column density from mass density (including pm1/pm2.5/pm10).
  • Added support for AEOLUS L1b/L2a/L2b products.
  • Added support for Aerosol and GHG CCI L2/L3 products.
  • Added support for GEOMS FTIR C2H2 ingestion.
  • Added support for OMI OMCHOCHO ingestion.
  • Added support for additional TES L2 products.
  • Added support for QA4ECV NO2 and HCHO products.
  • Added support for GOME, GOME-2, GOMOS, MIPAS, IASI, SCIAMACHY L1 products.
  • Added support for ODIN OSIRIS and SMR L2 products.
  • Changed --list-conversions command line option to --list-derivations. And added --list-derivations option to harpdump.
  • Use NC_64BIT_OFFSET when storing data to netcdf larger than 1GB.

Sustainable Development - Environmental Satellites - C
Published by svniemeijer about 8 years ago

harp -

📅 2016-10-16

  • Fixed wrong scaling factor in conversions related to the ideal gas law
    (10^2 vs. 10^-2) and barometric formula (10^-9 vs. 10^-6).
  • Fixed error in algorithm to derive gravity from latitude and height.
  • Fixed invalid sorting of collocation result file (which resulted in wrong
    results when performing a collocation resampling).
  • Fixed wrong independent dimension length for altitude/pressure bounds for
    SCIAMACHY Level2 ingestion.
  • Added apriori number density variables to SCIAMACHY Level2 ingestion.
  • Added variable conversion from latitude/longitude per vertical profile point
    to single (mid-point) latitude/longitude for profile.
  • Variable conversions for altitude and pressure bounds now also work when
    they are dependend on latitude and longitude dimensions.
  • Fixed crash in derived variable conversion for volume mixing ratio from
    mass mixing ratio (for total air).
  • Changed dimension argument of 'flatten()' operation from string to
    identifier.
  • Fixed issue where -inf/+inf valid_min/valid_max attributes could not be
    exported as float values in the Python harp.export_product() interface.
  • When deriving latitude/longitude bounds from mid points, latitude is now
    clamped to [-90,90] and longitude is wrapped to [-180,180].
  • Fixed volume mixing ratio unit for GEOMS-TE-UVVIS-DOAS-OFFAXIS ingestion.

Sustainable Development - Environmental Satellites - C
Published by svniemeijer over 8 years ago

harp -

📅 2016-06-30

  • The changes between 0.2 and 0.3 are extensive. The list below provides the
    most important aspects, but there are many more smaller changes that are not
    listed.
  • HARP 'actions' are now called 'operations' and are now executed in a strict
    sequence. This means that you can now do things like:
    'derive(var); var < 10; exclude(var)'
  • Added harpprofile command line tool that allows regridding and smoothing
    of vertical gridded data.
    Also added harp_product_regrid_vertical_with_axis_variable(),
    harp_product_regrid_vertical_with_collocated_dataset(), and
    harp_product_smooth_vertical() to C interface.
  • Added algorithm descriptions for the derived variable conversions to the
    HARP documentation.
  • Introduced a more strict definition of variable names for the HARP data
    format. As part of this the list of supported species and associated naming
    has also been revised.
  • Introduced strict separation between dry air and total/wet air. The default
    air density in HARP is now the total/wet air. Anything related to dry air
    should now be annotated as such in the variable name. As a result, 'dry_air'
    is now introduced as a species, and for mixing ratios the '_wet' postfix is
    removed and the '_dry_air' postfix is introduced.
    Derived variable conversions have been added for dry/wet conversion and for
    retrieving the needed 'molar mass' of total air.
  • Added functionality to deal with HARP Datasets and Product Metadata to the
    public C library interface. The collocation result interface in the C
    libraryinterface has also been reworked to now work with HARP Datasets.
  • HARP_ERROR_ACTION and HARP_ERROR_ACTION_SYNTAX have been renamed to
    HARP_ERROR_OPERATION and HARP_ERROR_OPERATION_SYNTAX.
    harp_product_execute_actions() has been renamed to
    harp_product_execute_operations().
  • Added operation to flatten data in a certain dimension and added
    harp_product_flatten_dimension().
  • Automatic detection of a product type when ingesting products now happens
    using CODA (using included codadef files with detection rules) for all
    products.
  • harp_import_global_attributes() has been removed (its functionality is
    now captured by the new Product Metadata interface).
  • Added public functions to allow raising/capturing of warnings in HARP:
    harp_report_warning(), harp_get_warning_handler(), harp_set_warning_handler()
  • Added public functions to print product/variable structure and content:
    harp_product_print(), harp_variable_print(), harp_variable_print_data()
  • Fixed issue in the HARP Python interface where using the value 0 for
    valid_min or valid_max did not work.
  • Added concatenate() function to HARP Python interface.
  • The 'include()' action/operation has been renamed to 'keep()'.
  • The derived variable conversions are now properly split according to the
    dimension types of a variable. This also solves a problem where certain
    valid conversions where not included in a '--list-conversions' print.
  • The derived variable conversions that produce uncertainty and covariance
    variables filled with NaNs from scratch have been removed.
  • Added derived variable conversions for gridded data (i.e. variables that
    depend on the latitude and longitude dimensions).
  • Removed variables flag_am_pm and flag_day_twilight_night and the conversions
    for flag_am_pm, flag_day_twilight_night, and flag_daytime. These conditions
    can all be determined by directly comparing against the solar angles.
  • Added support for ingestion of GOMOS L2, SCIAMACHY L2, ECMF GRIB and TEMIS
    assimilated ozone data.
  • Added 'species' ingestion option to the MIPAS L2 ingestion.
  • Added support for MLS CH3Cl, CH3CN, and CH3OH products.
  • Fixed issue in GEOMS ingestion where long variable names were truncated.
  • Introduced convention that dimensionless quantities should have an empty
    'units' attribute string value. Only non-quantities should omit the 'units'
    attribute.
  • Resolved incorrect correction for leap seconds in product ingestions.
  • Covariance variables have been renamed from '__cov' to '__covariance'.
  • The split of random/systematic for covariances has been removed.
    For ingested products that had such covariances, the random covariances are
    now retrieved as plain covariances and from systematic covariances only the
    trace is retrieved and provided as a systematic uncertainty.
  • Added 'geopotential' as a supported quantity with conversions from/to
    'geopotential height'. Added HARP_UNIT_GEOPOTENTIAL.
  • Renamed cloud optical thickness to cloud optical depth.
  • Added '-a/--operations' option to harpdump.
  • Support for HDF4 and HDF5 (for import/export) is now optional when building
    HARP, but still enabled by default.

Sustainable Development - Environmental Satellites - C
Published by svniemeijer over 8 years ago

harp - 0.2

📅 2016-06-30

  • Added Python interface.
  • CODA is no longer included as part of HARP, but is treated as a separate
    package, just like HDF4 and HDF5.
  • The CMake build system can now be used to build HARP on Windows and create
    a binary installer (using WIX).
  • Revised and renumbered error codes in C interface.
    Renamed HARP_ERROR_SCRIPT and HARP_ERROR_SCRIPT_SYNTAX error codes to
    HARP_ERROR_ACTION and HARP_ERROR_ACTION_SYNTAX. Changed HARP_ERROR_PRODUCT
    to either HARP_ERROR_INGESTION or HARP_ERROR_IMPORT depending on context.
    Added HARP_ERROR_EXPORT, HARP_ERROR_INVALID_PRODUCT and
    HARP_ERROR_INVALID_VARIABLE.
  • The HARP command line tools now link against the dynamic HARP library
    instead of the static version.
  • Updated embedded expat library to version 2.1.1.
  • Updated embedded udunits2 library to version 2.2.20.
    Also removed the HARP specific udunits XML configuration file since all
    HARP units are now included in udunits2.
  • Renamed all 'stdev' variables to 'uncertainty'.
  • Fixed parsing of dimension names; string dimension could be interpreted as
    independent dimension.
  • Improved the efficiency of the conversion from volume mixing ratio to number
    density and vice versa.
  • Update of S5P L2 ingestion module to the lastest version of the product
    format specification (S5P-KNMI-L2-0009-SD and relevant DLR PUMs).
  • Update of S5P L1B ingestion module to the lastest version of the product
    format specification (S5P-KNMI-L01B-0012-SD).
  • Added ingestion modules for the following product types:
    • Ozone CCI L2 limb profile
    • Ozone CCI L3 limb profile
    • Ozone CCI L2 nadir profile
    • Ozone CCI L3 nadir profile
    • Ozone CCI L4 nadir profile
    • Ozone CCI L2 total column
    • Ozone CCI L3 total column
  • Added public functions harp_get_fill_value_for_type(),
    harp_get_valid_min_for_type(), harp_get_valid_max_for_type(),
    harp_is_fill_value_for_type(), harp_is_valid_min_for_type(),
    harp_is_valid_max_for_type(), harp_ingest_test(),
    harp_doc_list_conversions(), harp_doc_export_ingestion_definitions(),
    harp_geometry_get_point_distance(), harp_geometry_has_point_in_area(),
    harp_geometry_has_area_overlap(), harp_basename(),
    harp_product_execute_actions(), harp_variable_set_description(),
    harp_variable_set_unit(), harp_variable_set_string_data_element(),
    harp_variable_verify(), harp_product_set_source_product(),
    harp_product_set_history(), harp_product_verify()
  • Added public functions for dealing with HARP collocations.
  • Moved defines for default harp units to harp.h.
  • Added operators =& and !& to the product actions to test integer variables
    against a bit mask, where a =& mask is equivalent to (a & mask) != 0 and
    a !& mask is equivalent to (a & mask) == 0.
  • Added functions to verify the internal consistency as well as compliance to
    HARP conventions of harp_product and harp_variable instances. Any instance
    of harp_product returned by harp_import() or harp_ingest() has been verified
    using these functions.
  • Added derived variable conversion from 'latitude {latitude}' (mid-points) to
    'latitude_bounds {latitude, independent}' (bounds); similar for longitude.
  • Updated HDF5 data format such that HDF5 products written by HARP are
    compatible with netCDF-4. HDF5 dimension scales are used to represent shared
    dimensions. The "dims" dataset attribute has been deprecated.
  • Changed netCDF-3/HDF4/HDF5 backends to be more explicit about implicitly
    created dimensions (which are used to store scalars and strings for formats
    that do not support this natively).
  • Fixed handling of scalars and empty strings in netCDF-3/HDF4/HDF5 backends.
  • Fixed unit attribute name in HARP products ('unit' -> 'units').
  • Changed all constants to be in line with CODATA 2014 as published by NIST.

Sustainable Development - Environmental Satellites - C
Published by svniemeijer almost 9 years ago

harp -

📅 2015-11-09

  • First beta release (using CODA 2.15).

Sustainable Development - Environmental Satellites - C
Published by svniemeijer over 9 years ago