Recent Releases of harp
harp -
📅 2025-03-16
-
The rebin() operation now takes into account existing weight variables
and properly handles angle variables. Rebinning the result of a
bin_spatial() operation to a lower resolution now gives the same result
as when a spatial bin was performed directly to that lower resolution. -
Added harp_convert_unit_double() and harp_convert_unit_float().
harp_convert_unit() is now deprecated. -
Performing a unit conversion on single precision floating point
variables will no longer convert the variable to double precision. -
Updates to product ingestions:
- GOME2 L2: added support for NHP/OHP ozone profile products and all
products now use float instead of double for variables. - GEOMS FTIR: reinterpret vmr from old GEOMS templates as dry air vmr.
- EarthCARE L1: add support for L1 ATLID, BBR, and MSI products.
- EarthCARE L2: all vertical profiles are inverted to make them
ascending, fixed geoid offset correction, and fixed AM__CTH_2B
ingestion. - S5P PAL L2: added support for Ocean Colour (L2__KD____) product.
- GOME2 L2: added support for NHP/OHP ozone profile products and all
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer about 1 month ago

harp -
📅 2024-11-11
-
Fixed appending non-float variables with different sub-dimensions when
using the Python harp.concatenate() function. -
Updates to product ingestions:
- ESACCI_GHG: fixed incorrect unit for CH4/CO variables
- GOME-2 L2: added support for O3MARS aerosol layer height and added
support for O3MARP V5 - S5P L2: added snow_ice_type/sea_ice_fraction ingestion for AAI/CO/CH4
- S5P L2 CH4: added 'corrected' option to ingest destriped columns and
fixed issue in NIR/SWIR selection of surface albedo - S5P PAL L2: added support for SRON/SICOR HDO (L2__HDO__S) product
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer 6 months ago

harp -
📅 2024-07-30
-
Fixed numpy 2.x compatibility issues in python interface.
-
Added derivations to retrieve dry air mass/volume mixing ratio from H2O
mass/volume mixing ratio with regard to dry air. -
Fixed some issues with building the HARP R interface extension.
-
Updates to product ingestions:
- S5P PAL L2 AOT: added support for cloud_fraction vs.
effective_cloud_fraction split
- S5P PAL L2 AOT: added support for cloud_fraction vs.
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer 9 months ago

harp -
📅 2024-06-12
- Updates to product ingestions:
- EARLINET: fixed memory issue when reading extinction data.
- EarthCARE: updates for new V11.2 format definition baseline
- GEOMS UVVIS.DOBSON: fixed issue with reading of int16 typed data.
- S5P PAL L2 SO2 COBRA: fixed cloud_fraction option behaviour and added
alternative qa value calculation for processor V02.00.00.
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer 11 months ago

harp -
📅 2024-03-15
-
Fixed incorrect weight division for random uncertainties in spatial binning.
-
Added check for unit equality when appending variables.
-
Fixed CMake build for locating Python 3.12.
-
Updates to product ingestions:
- EarthCARE: updates for new V11 format definition baseline
- ECMWF GRIB: added general support for new masterTablesVersion numbers
- S5P L2 CH4: added support for nir aerosol/cloud/surface property ingestion.
- S5P L2 CLOUD: added support for nir-band spatial grid data
- S5P PAL L2: added support for OClO
- S5P PAL L2 AOT: added support for V02.00.00 output format
- S5P PAL L2 BRO: fixed memory issue with reading total column and
uncertainty variables - S5P PAL L2 TCWV: fixed wrong quantity for density variables
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer about 1 year ago

harp -
📅 2023-11-13
-
Fill values of floating point types are now explicitly set to NaN when
exporting to a netcdf or hdf5 file. -
Updates to product ingestions:
- EARLINET: fixed issue where no backscatter/extinction data was ingested.
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer over 1 year ago

harp -
📅 2023-10-12
- Updates to product ingestions:
- IASI L1C: added missing angle variables
- S5P PAL L2 AER_OT: added missing wavelength variable
- S5P PAL L2 CHOCHO: fixed missing units for number density variables
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer over 1 year ago

harp -
📅 2023-09-19
-
Updates to product ingestions:
- GEOMS FTIR: added support for C2H3NO5/PAN and C2H4, added support for
template V3. - GEOMS TCCON: added support for template V6.
- S5P L2 ALH: add support for revised surface_albedo variable and added
surface_albedo_uncertainty. - S5P L2 O3: fix custom V1 QA filter for NRTI products
- S5P PAL L2: fixed memory issue when reading aerosol_optical_depth_validity
of AER_OT product.
- GEOMS FTIR: added support for C2H3NO5/PAN and C2H4, added support for
-
Fixed missing documentation on weight factor handling in binning algorithm.
-
Updated embedded expat library to version 2.5.0.
-
Fixed issue where jpeg dll was missing from windows binary installer.
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer over 1 year ago

harp -
📅 2023-07-24
-
The python harp.import_product() function now allows HARP .pth files to be
provided as input. -
Reintroduced propagation of *_uncertainty_random variables when performing
spatial binning with area weighted averaging. -
Allow regridding using a multi-dimensional source axis-variable as input.
e.g. convert from an 'altitude {latitude,longitude,vertical}' grid to
a fixed 'altitude {vertical}` grid. -
Added documentation for regridding/binning algorithms.
-
Updates to product ingestions:
- ACE-FTS: fix ingestion of ASCII formatted products
- GOME-2 L2: added support for offline absorbing aerosol index (O3MARP)
- EARLINET: added support for new format (removed support for old format)
- EarthCARE: added support for AM__ACD_2B, AM__CTH_2B, BM__RAD_2B,
BMA_FLX_2B, MSI_AOT_2A, MSI_CM__2A, and MSI_COP_2A - ECMWF GRIB: added support for new masterTablesVersion=30 data
- S5P PAL L2: added support for AER_OT, BRO, CHOCHO, SIF, SO2 COBRA, and
TCWV
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer almost 2 years ago

harp -
📅 2023-06-08
-
Rename all uses of 'water vapour' to 'water vapor'.
-
Added =| bitfield operater to allow filtering for the case where only some
bits within the mask are set. -
The uncertainty propagation of *_uncertainty_random variables for the
temporal binning operation now always assumes no correlation. For spatial
binning the random component is removed for the area weighted average
approach and it is propagated using no correlation for the simple binning
based on center points. -
Added derivation for 'area {latitude,longitude}' and fixed area calculation
where South Pole was an edge point. -
Fixed reading of string arrays in HARP Python interface.
-
Updates to product ingestions:
- ECMWF GRIB: added support for new masterTablesVersion=29 data
- S5P L2 CLOUD: fixed qa_value ingestion for CRB (now use qa_value_crb)
- S5P L2 SO2: added support for SO2 layer height
- OMI L3: added support for OMHCHOd, OMNO2d, and OMSO2e
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer almost 2 years ago

harp -
📅 2022-12-12
-
Added additional conventions for variable names related to areosol/cloud
properties. -
Fixed issue with geographical area overlap calculation and improvided
performance of performing operations on geographical polygon data. -
Windows .msi installer now allows HARP component selection for core, Python,
IDL, MATLAB, and R. -
Updates to product ingestions:
- AEOLUS L2B: properly split datetime, datetime_start, and datetime_stop
- CCI L3 O3 LP: fixed wrong variable name
- EarthCARE: added additional product types and updated ACM_CAP_2B
- ECMWF GRIB: changed RWC/SWC/LWC/IWC variable names
- OSIRIS L2: fixed wrong variable names
- TES TL2ATMTN/TL2ATMTL: fixed wrong variable names
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer over 2 years ago

harp -
📅 2022-10-31
-
Improved algorithm to calculate center point from latitude/longitude bounds
polygon. -
Add derivation of time dependent from time indepent for datetime variables.
-
Updates to product ingestions:
- GEOMS PANDORA: added systematic/random uncertainty for effective
temperature, renamed H2CO in harp variable names to HCHO. - OCO: added support for ACOS/OCO-2/OCO-3 L2/L3 products.
- Ozone CCI L3 LP: fixed memory issue when reading
merged_ozone_concentration. - S5P L2 HCHO: added amf=clear_sky ingestion option
- GEOMS PANDORA: added systematic/random uncertainty for effective
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer over 2 years ago

harp -
📅 2022-05-18
- Updates to product ingestions:
- CLOUDNET L2 classification: fixed fillvalue mapping of values to NaN.
- GEOMS PANDORA: fixes for template V003 support
- GEOMS FTIR: allow instrument lat/lon/alt variables to be one-dimensional
with replicated values. - GEOMS UVVIS.BREWER: added random+systematic uncertainty variables.
- S5P L2 CO: fixed vertical ordering of new apriori profile.
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer almost 3 years ago

harp -
📅 2022-03-14
-
Fixed issue where bin() operation was removing scalar enumeration variables.
-
Updates to product ingestions:
- ECMWF GRIB: fixed parameter mapping for CAMS model data that uses
masterTablesVersion=27 data and added support for new
masterTablesVersion=28 data - GEOMS UVVIS.BREWER: added support for template V001
- GEOMS PANDORA: added support for template V003
- S5P L2 AAI: added new 335/367 wavelength pair
- S5P L2 CO: added new apriori profile and support for new AVK unit
- S5P L2 O3 Profile: added surface_albedo and cloud_albedo
- S5P L2 HCHO/O3/SO2: added support for updated NRTI indication attribute.
- ECMWF GRIB: fixed parameter mapping for CAMS model data that uses
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer about 3 years ago

harp - 1.14
📅 2021-10-20
-
Updated embedded udunits2 library to version 2.2.28.
-
Improved CMake build system.
-
Fixed issue when building against HDF5 1.12.
-
Updates to product ingestions:
- AEOLUS L1B: fixed ingestion of Mie/measurement data; added sensor angles
- AEOLUS L2B: fixed ingestion of Mie data; added sensor angles
- CLOUDNET L2 classification: added support for new (netcdf4) format
- EARTHCARE: added very early support (limited version of ECA_ACM_CAP_2B)
- ECMWF GRIB: fixed unit of backscatter coefficient 1/(milli sr) -> 1/(m sr),
added support for new masterTablesVersion=27 data - GEOMS UVVIS.DOAS: allow template V007 files to be stored in hdf5 format
- GHG CCI: added support for TROPOMI WFMD L2
- OMI L2: added sensor alt/lat/lon and surface altitude/pressure
- OMI L2 OMAERUV: fix ingestion issues
- QA4ECV HCHO: added amf=clear_sky ingestion option
- S5P L2 HCHO: renamed cloud_altitude variable to cloud_height
- S5P L2 NO2: added retrieval of O2-O2 cloud (O22CLD) data
- S5P L2 NO2+FRESCO: added scene_albedo and scene_pressure
- S5P L2 O3 Profile: fixed wrong concentration variables (vmr -> nd)
- S5P L2 SO2: updated SO2_type enumeration list
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer over 3 years ago

harp -
📅 2021-03-12
-
Added rebin() operation.
-
Added harp_product_rebin_with_axis_bounds_variable() function.
-
Added support for datetime_bounds variable.
-
Fixed issue in =& operation (it returned true if any of the bits were 1
instead of all bits). -
Updates to product ingestions:
- GEOMS: renamed all site_name variables to location_name
- GEOMS FTIR H2CO: fixed issue in determining unit for H2O data (that would
fail the ingestion) - S5P L2 CO: added co=corrected ingestion option
- SCIAMACHY L2: fixed averaging of cloud fraction for co-added pixels
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer about 4 years ago

harp -
📅 2020-10-29
-
The csv output of 'harpdump --dataset' can now be used as content of a .pth
file. This improves the performance of harpcollocate/harpmerge/etc. as they
no longer need to open and extract metadata from files themselves anymore. -
harpcollocate now stores differences as signed values in collocation result
file (instead of absolute differences). -
Added harp_dataset_prefilter(). This function is now also called within
harpmerge and harpcollocate to pre-filter the input dataset(s) for datetime
and collocate left/right filters that are at the start of the operations
string. -
Added format and history fields to harp_product_metadata.
-
Added harp.import_product_metadata() to Python interface.
-
harp_dataset_add_product() now ignores files/directories starting with '.'.
-
Added uncertainty propagation to bin() and bin_spatial() operations
(using uncorrelated assumption for bin() and correlated for bin_spatial()). -
Added global option to determine whether uncertainty propagation should
use a correlated or uncorrelated approach. Currently only used for the bin()
operation. -
Added global option to create a collocation_datetime variable when performing
a collocate_left() or collocate_right() operation. -
Added wildcard pattern matching ('*'/'?') to keep() and exclude() operations.
-
Added derivation to retrieve angstrom exponent from aerosol optical depth.
-
Added derivation to retrieve tropospheric/stratospheric column avk from
total column avk and tropopause altitude. -
Fixed potential memory issues and improved overall code stability.
-
Updates to product ingestions:
- S5P L2 CLOUD: added cloud_type and cloud_top_temperature
- S5P L2 O3: added qa_filter=custom ingestion option to set validity based
on official PRF recommendation for processor V1.x generated products - MLS L2: fixed memory issue in reading of validity variable and updated
validity flag mapping to version 4.x-4.0 of the Data Quality document
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer over 4 years ago

harp -
📅 2020-05-19
-
Added derivations to retrieve column number density from column vmr.
-
Fixed issue in bin_spatial() where the weightfactor could be too high when
oversampling the input data. Grid cells that intersected with the data
polygon could end up with a weightfactor >1.0 (but <2.0). -
Fixed issue where molar mass was incorrectly provided for the species with
the shortest matching name. e.g. the molar mass of CO was returned for CO2
and of NO for NO2. This impacts vmr <-> mmr conversions for BrO2, COF2, COS,
CO2, C2H2O2, H2O2, NOCl, NO2, N2O, N2O5, and all isotopologues (e.g. H2O_161). -
Removed irrelevant harp_str64() and harp_str64u() functions (HARP does not
support 64bit integers). -
Fixed issue where molar mass linked to variables with a 'surface',
'tropospheric', or 'stratospheric' prefix returned 0. -
Fixed error/crash in deriving dry air column vmr from dry air column mmr.
-
Updates to product ingestions:
- GOME-2 L2: added support for SO2 columns at all plume heights
- OMI L2 CLDO2/CLDRR: added validity variable
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer almost 5 years ago

harp -
📅 2020-03-31
-
Added support for -ar/--reduce-operations parameter in harpmerge and
for reduce_operations parameter in harp.import_product() Python function. -
Fixed incorrect "invalid variable" error on weight variables for angular
quantities when performing a harpmerge. -
Fixed issue where the udunits2 library could not be initialized if the
environment had a LC_NUMERIC locale that did not have '.' as decimal
separator. -
Fixed issue in name mangling of embedded expat library.
-
Updates to product ingestions:
- GOMOS L1 Limb: fixed unit of wavelength_photon_radiance (and uncertainty)
and documented conversion algorithm - S5P L2 O3 PR: support for IODD 12.0.0
- S5P L2 O3 TPR: removed support (future availability of product uncertain)
- GEOMS PANDORA: allow for trailing \0 in global attribute values
- GOMOS L1 Limb: fixed unit of wavelength_photon_radiance (and uncertainty)
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer about 5 years ago

harp -
📅 2019-12-01
-
Fixed issue where using the 'bin()' operation did not correctly create a
'<variable>_weight' variable from a 'weight' variable when it was missing. -
Added default units to documention on variable name conventions.
-
Updates to product ingestions:
- Calipso L2: fixed unit of backscatter coefficient
- ERS GOME L2: fixed unit of cloud top height
- S5P L2 AER_LH: fixed unit of aerosol optical depth
- S5P L2 SO2: add scaled avk for 1km/7km/15km box profiles
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer over 5 years ago

harp -
📅 2019-11-01
-
Fixed issue where results from deriving partial column densities from vmr and
pressure were a factor 10 too low. -
Fixed issue where operation following an index filter operation did not
get executed.
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer over 5 years ago

harp -
📅 2019-10-21
-
Added operations to filter on dimension indices (e.g. 'index(vertical) < 5').
-
Changed bin() and bin_spatial() operations to use/produce separate 'count'
and 'weight' variables. The counts/weights are used to allow the following
commands to now be equivalent:
harpmerge -a 'bin()' -ap 'bin()' a.nc b.nc merged.nc
harpmerge -ap 'bin()' a.nc b.nc merged.nc
As well as the following:
harpmerge -a 'bin_spatial(...)' -ap 'squash(time, (latitude_bounds, longitude_bounds));bin()' a.nc b.nc merged.nc
harpmerge -ap 'bin_spatial(...)' a.nc b.nc merged.nc
-
Fixed issue with invalid bounding box optimization for spherical area
filter operations. -
Fixed missing error handling when using invalid operations in
harp.export_product() in Python. -
Fixed Python3 compatibility issue when exporting string data.
-
Updates to ECMWF GRIB, IASI L1/L2, GOME-2 L1/L2, MIPAS L1, and S5P L2
product ingestions.
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer over 5 years ago

harp - 1.8
📅 2019-07-31
-
Added R interface.
-
Improved derivation for tropopause altitude/pressure (to not return NaN).
-
Added derivations for gravity.
-
Added derivations to convert between total air and dry air mixing ratios.
-
Fixed valid() operation to properly use valid_min/valid_max attribute values.
-
Fixed issue in adding *_count variable in append/merge when count variable
had more than one dimension. -
Updates to ECMWF GRIB, GEOMS MWR, and S5P L2 product ingestions.
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer over 5 years ago

harp -
📅 2019-06-06
-
Added --no-history option to harpconvert, hardump, and harpmerge.
-
Added harp.execute_operations() and harp.convert_unit() to Python interface.
-
Added optional min_collocation_index/max_collocation_index arguments to
collocate_left()/collocate_right() operations. -
harp_product_set_history() and harp_product_set_source_product() now accept
NULL as argument to clear the current value. -
Appending/merging products now automatically adds missing *_count variables.
-
Fixed potential crash during initialization of HARP ingestion module.
-
Fixed invalid results when using bin_spatial with latitude_bounds {time,2}
and longitude_bounds {time,2} variables (i.e. range instead of polygon). -
Fixed array ordering in MATLAB interface. Multi-dimensional arrays are now
represented from fastest running to slowest (=fortran ordering) as intended. -
Update to S5P L2 product ingestion.
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer almost 6 years ago

harp -
📅 2019-04-29
-
Added clamp() and squash() operations.
-
Allow bin() and sort() operations to be performed using a set of variables.
-
harp_product_sort() now takes a list of variables.
-
The rename() operation no longer fails if the source variable does not exist
but a variable with the target name already exists. -
The python harp.import_product() function now uses the C library to perform
the contactenation of products. -
Added post_operations argument to python harp.import_product() function.
-
Besides using a polygon of >=3 vertices, a geospatial area can now also be
defined via a bounding rect. Use e.g. 'latitude_bounds {time,2}' and
'longitude_bounds {time,2}' to define the min/max of the bounding rect. -
Added derivation for tropospheric/stratospheric column from partial column
profile and tropopause. -
Corrected derivation for calculating tropopause altitude/pressure.
-
Deriving a variable will now use the conversion that requires the minimum
amount of intermediate variables to be calculated. -
Added support for scene albedo/pressure/type variables.
-
Fixed issue where regridding in the vertical dimension using pressure as axis
could end up with wrong pressure_bounds values. -
Fixed problem in matlab mex extension determination on linux when using the
configure script. -
Fixed setting of default CODA definition path and udunits2 xml path for
Python on Windows. -
Added format, variable count, and dimension sizes to harpcheck output.
-
Added support for TCCON products in GEOMS format.
-
Updates to AEOLUS L2A, ESACCI_OZONE_L2_TC, GEOMS FTIR, GOME2 L2, OMI L2 NO2,
QA4ECV, and S5P L2 product ingestions. -
Updated embedded expat library to version 2.2.6.
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer almost 6 years ago

harp - 1.5
📅 2018-12-17
-
Fixed issue where vertical smoothing/regridding provided invalid results if
input variables were available as float instead of double data. -
harp_product_get_derived_variable() no longer converts data to double if
no data type was provided and data already had the right unit. -
Fixed issue where listing derivations for a specific variable would not show
all possible conversions. -
Add derivation for cloud height from cloud base/top height.
-
Added support for Dobson, Pandora and Sonde products in GEOMS format.
-
Updates to ACE-FTS, GOME/GOME-2 L2, and S5P L2 product ingestions.
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer over 6 years ago

harp -
📅 2018-09-28
-
Added -t/--target option to harpdump --list-derivations.
-
Added variable_name parameter to harp_doc_list_conversions().
-
Added support for meridional/zonal wind speed, and planetary boundary layer
height variables. -
Added support for sic (Shannon information content) variables for vertical
profiles. -
Aerosol optical depth and extinction coefficient variables will now
automatically use a log/log regridding in the spectral dimension.
This is equivalent to using the Angstrom Exponent approach. -
Fixed interpolation issue where points beyond the source grid could end up
having an arbitrary wrong value (instead of NaN). This is a regression of
harp 1.3. -
Add derivations between wind speed/direction and zonal/meridional velocity.
-
Flattening a dimension now special cases a dimension length 1 (with less
modifications to variables/product). -
Fixed issue where a point distance of 0 could sometimes result in NaN.
-
Updates to S5P L2, GOME/GOME-2 L2, GEOMS UVVIS-DOAS-OFFAXIS, ECMWF GRIB, and
AERONET product ingestions.
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer over 6 years ago

harp -
📅 2018-07-05
-
Added bin_spatial() operation.
-
Improved binning operation to automatically filter for NaN values and add
variable-specific count variables to specify the number of non-NaN values. -
Performing a binning operation on top of a binning operation will propagate
averages based on the available count variables (i.e. as if the final bin was
performed on the original data). -
Entries added to the global 'history' attribute now also include the current
utc time and harp version before the execution command. -
Allow NULL as argument for harp_variable_set_description() and
harp_variable_set_unit() to clear the current description/unit. -
Added harp_product_bin_spatial() and harp_variable_copy_attributes()
functions. -
Added optional parameter to regrid() operation to specify the axis boundaries.
-
Python harp.import_product() now ignores empty products when ingesting a list
of products. -
Added derivations:
- column vmr/mmr from column density ratios
- total column mass density from surface pressure and pressure profile
- mass mixing ratio from mass density
- between dry air and total air mass (column) density
- between dry air, total air, and H2O number densities
-
Fixed derivation of altitude from gph/latitude.
-
Updates to S5P L2 and MIPAS L2 product ingestions.
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer almost 7 years ago

harp -
📅 2018-04-09
-
Allow data compression for HDF5 output:
- Added harp_set_option_hdf5_compression() and
harp_get_option_hdf5_compression() functions to C library. - Added --hdf5-compression option to harpconvert and harpmerge.
- Added hdf5_compression parameter to harp.export_product() Python function.
- Added harp_set_option_hdf5_compression() and
-
Allow regrid() operation to interpolate in the time dimension.
-
Improved binning of angle values (average of 1 and 359 [deg] is now 0).
-
Added bin() operation variant without parameters.
-
The bin() operation using a collocation result file now retains the
collocation_index variable (with the first value for each bin). -
Changed smooth() operation such that each vertical level where the source
grid point was NaN the target grid point also remains NaN. -
Added -ap option to harpmerge.
-
Added --dataset option to harpdump and changed output format of
harp_dataset_print() and harp_product_metadata_print() to csv format. -
Added 'operations' parameter to Python harp.export_product() function.
-
Importing/exporting a product with operations in Python now automatically
updates the history attribute. -
Fixed issue where harpmerge did not introduce a time dimension for all
variables. -
Fixed issue where datetime_start and datetime_stop global attributes were not
created when exporting in HDF4 or HDF5 format. -
String attributes in HDF5 files can now be empty strings (using H5S_NULL).
-
Allow HARP to import HDF5 files that are generated by the netCDF4 library
(allow numeric attributes to be 1-element arrays instead of scalars). -
harpcheck now returns with exit code 1 if any of the imports fail.
-
harpcollocate --update now only allows one dataset as argument.
-
Fixed performance issue with spatial filtering on systems with slow math lib.
-
Fixed mapping description of GOME2 L1 product ingestion.
-
Fixed wrong results of area overlap calculation.
-
Replaced '_' by '-' in long versions of argument names of harpcollocate.
-
Added support for dfs (degrees of freedom for signal) profiles.
-
Updates to S5P L2 product ingestions.
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer about 7 years ago

harp - 1.1
📅 2017-11-06
-
Parameters that required a file or a directory as dataset reference can now
also take a file with a .pth extension as input. This file should contain a
line-separated list of paths to files and/or directories of the dataset. -
Specifying a directory for a dataset will now recursively add all products
(including sub-directories and contents of .pth files). -
The variable that is binned on with a 'bin(variable)' operation is now
always kept in the resulting product. -
The bin() operation can now handle 'count' variables (summing it if present
and creating it otherwise). -
Fixed issue where collocate filter operation on a product with a single
sample could end in an infinite loop. -
Fixed issue in harpcollocate when providing a -nx option without a -ny.
-
Fixed issue with wrap() operation for values < specified minimum.
-
Fixed issue with collocate_left/collocate_right operations not including
last sample when applying it on products in an external format. -
harpcollocate now ignores empty products instead of raising errors for them.
-
Added -oa/-ob and -aa/-ab options to harpcollocate matchup.
-
Added solar_declination_angle and solar_hour_angle variables and derivations.
-
Added orbit_index variable and added this to various product ingestions.
-
Fixed derivations of solar azimuth/zenith angles from datetime and lat/lon.
-
Added derivations for averaging kernel conversions.
-
Added derivations for tropopause altitude/pressure.
-
Added NH3 (amonia) as supported species.
-
Updates to ECMWF-GRIB, GEOMS-TE-UVVIS-DOAS-OFFAXIS, GEOMS-TE-FTIR, and
S5P L2 product ingestions (and several others for the new orbit_index).
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer over 7 years ago

harp - 1.0
📅 2017-09-25
-
The import and ingest functions have been combined into a single import
function that can both read HARP compliant products as well as ingest from
the supported external formats. This means:- harp_import() and harp_ingest() are combined into a single harp_import()
- harp_ingest_test() has been renamed to harp_import_test()
- harpfilter has been removed (use harpconvert instead)
- '--test' option of harpconvert has been removed (use harpcheck instead)
-
harpcollocate has been completely rewritten.
- any 1D time-dependent variable can now be a collocation criterium
- nearest neighbour filtering with weighted average is no longer supported
- collocation result files are not backward compatible due to a change in
the CSV header - arguments to harp_collocation_result_new() and
harp_collocation_result_add_pair() have changed
-
Added support for enumeration variables. Enumeration variables have a int8,
int16, or int32 type and provide an index into a fixed list of enum names.
Enumeration variables can be treated as string values in operations. -
Added MATLAB and IDL interfaces for HARP.
-
Added -o option to harpdump and harpmerge.
-
Added -a and -o options for 'harpdump --list-derivations'.
-
'--list-derivations' option of harpconvert has been removed (use harpdump
instead). -
Added -l option to harpmerge.
-
Renamed area/point operations:
- area_mask_covers_area -> area_covers_area
- area_mask_covers_point -> point_in_area
- area_mask_intersects_area -> area_intersects_area
- point_in_area -> area_covers_point
The first part of the operation name now reflects the item in the
product and the second part reflects the parameter that is provided.
-
Improved performance of point/area vs. area overlap operations by first
comparing against the bounding box of a polygon. -
Added sort() and area_inside_area() operations.
-
The derive() operation can now be used without dimension specification
to perform a unit and/or data type conversion on an existing variable. -
The derive_smoothed_column(), regrid() and smooth() operations can now
use a single merged collocated product (instead of a combination of a csv
collocation result file and a dataset directory of collocated products). -
An area parameter for operations can now either be a reference to an
area mask file or a single polygon provided as lat/lon array parameters. -
Added derivation to derive the area (size) of latitude/longitude bounds.
-
Added derivation to derive degree of freedom for signal (DFS) from 2D
averaging kernels (AVK). -
Derivations of bound variables will now clamp to the edge midpoints when
the 'regrid_out_of_bounds' option is set to 'edge'. -
The 'data_type' parameter for harp_product_get_derived_variable() and
harp_product_add_derived_variable() is now a pointer to allow it to be
optional (i.e. pass NULL to use the default data type). -
Added harp_geometry_get_area(),
harp_product_get_smoothed_column_using_collocated_product(),
harp_product_smooth_vertical_with_collocated_product(),
harp_product_sort(), harp_product_regrid_with_collocated_product(),
harp_set_udunits2_xml_path(), harp_set_udunits2_xml_path_conditional(),
and harp_variable_set_enumeration_values() functions. -
harp_import_product_metadata() and harp_dataset_import() now have an
'options' parameter. -
Changed 'percentage' parameter of harp_geometry_has_area_overlap() to
'fraction'. -
harp_product_set_source_product() now has a product path as parameter
and stores the base name of this as source_product value. -
Added HARP_UNIT_AREA default unit.
-
The C interface for collocation results was modified to support a
dynamic set of collocation criteria differences (based on variable names
for these differences). -
The Python interface now sets the udunits2 xml path dynamically,
relative to the location of the HARP C library. -
wrap() operation now properly updates valid_min/valid_max properties.
-
codadef files are now installed in /share/coda/definitions.
This allows a present CODA installation to use the same codadef files. -
Windows installation now installs data and documentation files in the
subdirectory /share/harp. -
Fix issue in deriving total column from partial column profile if partial
column profile did not have {time,vertical} dimensions. -
Fix invalid results from binning operation/function.
-
Improve speed of --list-derivations on a product.
-
Split data format documentation into seperate sections under a new
heading 'conventions'. -
Added product ingestions for Cloudnet L2a classification products.
-
Updates to ECMF-GRIB, GEOMS-TE-UVVIS-DOAS-OFFAXIS, MIPAS, QA4ECV, and
S5P product ingestions.
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer over 7 years ago

harp - 0.6
📅 2017-04-23
-
Added ingestion support for ACE-FTS, CALIPSO, Cloud CCI, and NPP-Suomi.
-
Added bin() and set() operations.
-
Added optional data_type parameter to derive() operation.
-
Major refactoring of internal implementation of operations (greatly
reducing and simplifying the code). -
Fix application of apriori for smoothed column derivation (for vertical
range where partial columns are NaN, don't subtract apriori*AVK, but just
set to apriori). -
Fix memory issue in partial column from vmr dry air derivation.
-
Remove double definition of radiance conversions.
-
Added global HARP option
regrid_out_of_bounds
. -
Added harp_product_bin() function.
-
Add data_type parameter to harp_product_add_derived_variable() and
harp_product_get_derived_variable() functions. -
Added data_type parameter to harp_product_get_derived_variable() and
harp_product_add_derived_variable() functions. -
Allow 'Conventions' global attribute to contain more than just a reference
to HARP. -
Renamed O3MSAF codadef to ACSAF. When updating HARP, you should remove
any existing O3SMAF-*.codadef files from your installation directory.
Otherwise you will get 'detection rule for ... is shadowed by ... ' errors. -
Fixed issue with TES TL2CH4L ingestion.
-
Updates to Aeolus, S5P, QA4ECV, and Ozone CCI product ingestions.
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer about 8 years ago

harp -
📅 2017-03-26
- Changed Open Source licence of HARP from GPLv2 to BSD (3-Clause).
- Changed all '-' to '_' in operation function names to make them identifiers.
- Added harpmerge command line tool.
- Added regrid(), rename(), smooth(), derive_smoothed_column(), point_in_area(), and wrap() operations.
- Removed the harpprofile command line tool. Its functionality is replaced by the regrid() and smooth() operations.
- Renamed harp_product_regrid_vertical_with_collocated_dataset() to harp_product_regrid_with_collocated_dataset() and harp_product_smooth_vertical() to harp_product_smooth_vertical_with_collocated_dataset()
- Added vertical_unit parameter to harp_product_smooth_vertical() and harp_product_regrid_with_collocated_dataset() functions.
- Replaced harp_product_regrid_vertical_with_axis_variable() with harp_product_regrid_with_axis_variable(). It can now be used for other dimensions too (except the time dimension).
- Added harp_collocation_result_filter_for_collocation_indices(), harp_product_get_smoothed_column(),
harp_product_get_smoothed_column_using_collocated_dataset(), harp_product_remove_variable_by_name(), harp_variable_append(), harp_product_append(), and harp_variable_smooth_vertical() functions. - Renamed harp_product_get_variable_id_by_name() to harp_product_get_variable_index_by_name().
- Python harp.import_product() and harp.ingest_product() functions now accept a list of filenames or a globbing pattern as parameter (which will return the result of harp.concatenate() on the imported/ingested products).
- Automatically set CODA definition path for HARP Python interface (path can be overruled with CODA_DEFINITION environment variable).
- The parsing of the operations string has been rewritten using flex/yacc. This has fixed several issues related to the parsing and execution of operations.
- Fixed issue with flatten() operation providing wrong results.
- Fixed issue where vertical regridding to a common grid provided wrong results.
- Fixed issue where interval interpolation for a pressure grid was not using a log-linear interpolation.
- Aligned order of latitude/longitude arguments in all functions to have latitude first and longitude second.
- Added unit 'ppv' to the included udunits2-common.xml file.
- Added variables for surface temperature, total water vapour content, and total cloud cover to ECMWF GRIB ingestion.
- The default internal units have been changes to SI (except for molar mass). All algorithms have been adapted accordingly.
- Added variable conversions for: apriori profiles, column density from mass density (including pm1/pm2.5/pm10).
- Added support for AEOLUS L1b/L2a/L2b products.
- Added support for Aerosol and GHG CCI L2/L3 products.
- Added support for GEOMS FTIR C2H2 ingestion.
- Added support for OMI OMCHOCHO ingestion.
- Added support for additional TES L2 products.
- Added support for QA4ECV NO2 and HCHO products.
- Added support for GOME, GOME-2, GOMOS, MIPAS, IASI, SCIAMACHY L1 products.
- Added support for ODIN OSIRIS and SMR L2 products.
- Changed --list-conversions command line option to --list-derivations. And added --list-derivations option to harpdump.
- Use NC_64BIT_OFFSET when storing data to netcdf larger than 1GB.
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer about 8 years ago

harp -
📅 2016-10-16
- Fixed wrong scaling factor in conversions related to the ideal gas law
(10^2 vs. 10^-2) and barometric formula (10^-9 vs. 10^-6). - Fixed error in algorithm to derive gravity from latitude and height.
- Fixed invalid sorting of collocation result file (which resulted in wrong
results when performing a collocation resampling). - Fixed wrong independent dimension length for altitude/pressure bounds for
SCIAMACHY Level2 ingestion. - Added apriori number density variables to SCIAMACHY Level2 ingestion.
- Added variable conversion from latitude/longitude per vertical profile point
to single (mid-point) latitude/longitude for profile. - Variable conversions for altitude and pressure bounds now also work when
they are dependend on latitude and longitude dimensions. - Fixed crash in derived variable conversion for volume mixing ratio from
mass mixing ratio (for total air). - Changed dimension argument of 'flatten()' operation from string to
identifier. - Fixed issue where -inf/+inf valid_min/valid_max attributes could not be
exported as float values in the Python harp.export_product() interface. - When deriving latitude/longitude bounds from mid points, latitude is now
clamped to [-90,90] and longitude is wrapped to [-180,180]. - Fixed volume mixing ratio unit for GEOMS-TE-UVVIS-DOAS-OFFAXIS ingestion.
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer over 8 years ago

harp -
📅 2016-06-30
- The changes between 0.2 and 0.3 are extensive. The list below provides the
most important aspects, but there are many more smaller changes that are not
listed. - HARP 'actions' are now called 'operations' and are now executed in a strict
sequence. This means that you can now do things like:
'derive(var); var < 10; exclude(var)' - Added harpprofile command line tool that allows regridding and smoothing
of vertical gridded data.
Also added harp_product_regrid_vertical_with_axis_variable(),
harp_product_regrid_vertical_with_collocated_dataset(), and
harp_product_smooth_vertical() to C interface. - Added algorithm descriptions for the derived variable conversions to the
HARP documentation. - Introduced a more strict definition of variable names for the HARP data
format. As part of this the list of supported species and associated naming
has also been revised. - Introduced strict separation between dry air and total/wet air. The default
air density in HARP is now the total/wet air. Anything related to dry air
should now be annotated as such in the variable name. As a result, 'dry_air'
is now introduced as a species, and for mixing ratios the '_wet' postfix is
removed and the '_dry_air' postfix is introduced.
Derived variable conversions have been added for dry/wet conversion and for
retrieving the needed 'molar mass' of total air. - Added functionality to deal with HARP Datasets and Product Metadata to the
public C library interface. The collocation result interface in the C
libraryinterface has also been reworked to now work with HARP Datasets. - HARP_ERROR_ACTION and HARP_ERROR_ACTION_SYNTAX have been renamed to
HARP_ERROR_OPERATION and HARP_ERROR_OPERATION_SYNTAX.
harp_product_execute_actions() has been renamed to
harp_product_execute_operations(). - Added operation to flatten data in a certain dimension and added
harp_product_flatten_dimension(). - Automatic detection of a product type when ingesting products now happens
using CODA (using included codadef files with detection rules) for all
products. - harp_import_global_attributes() has been removed (its functionality is
now captured by the new Product Metadata interface). - Added public functions to allow raising/capturing of warnings in HARP:
harp_report_warning(), harp_get_warning_handler(), harp_set_warning_handler() - Added public functions to print product/variable structure and content:
harp_product_print(), harp_variable_print(), harp_variable_print_data() - Fixed issue in the HARP Python interface where using the value 0 for
valid_min or valid_max did not work. - Added concatenate() function to HARP Python interface.
- The 'include()' action/operation has been renamed to 'keep()'.
- The derived variable conversions are now properly split according to the
dimension types of a variable. This also solves a problem where certain
valid conversions where not included in a '--list-conversions' print. - The derived variable conversions that produce uncertainty and covariance
variables filled with NaNs from scratch have been removed. - Added derived variable conversions for gridded data (i.e. variables that
depend on the latitude and longitude dimensions). - Removed variables flag_am_pm and flag_day_twilight_night and the conversions
for flag_am_pm, flag_day_twilight_night, and flag_daytime. These conditions
can all be determined by directly comparing against the solar angles. - Added support for ingestion of GOMOS L2, SCIAMACHY L2, ECMF GRIB and TEMIS
assimilated ozone data. - Added 'species' ingestion option to the MIPAS L2 ingestion.
- Added support for MLS CH3Cl, CH3CN, and CH3OH products.
- Fixed issue in GEOMS ingestion where long variable names were truncated.
- Introduced convention that dimensionless quantities should have an empty
'units' attribute string value. Only non-quantities should omit the 'units'
attribute. - Resolved incorrect correction for leap seconds in product ingestions.
- Covariance variables have been renamed from '__cov' to '__covariance'.
- The split of random/systematic for covariances has been removed.
For ingested products that had such covariances, the random covariances are
now retrieved as plain covariances and from systematic covariances only the
trace is retrieved and provided as a systematic uncertainty. - Added 'geopotential' as a supported quantity with conversions from/to
'geopotential height'. Added HARP_UNIT_GEOPOTENTIAL. - Renamed cloud optical thickness to cloud optical depth.
- Added '-a/--operations' option to harpdump.
- Support for HDF4 and HDF5 (for import/export) is now optional when building
HARP, but still enabled by default.
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer over 8 years ago

harp - 0.2
📅 2016-06-30
- Added Python interface.
- CODA is no longer included as part of HARP, but is treated as a separate
package, just like HDF4 and HDF5. - The CMake build system can now be used to build HARP on Windows and create
a binary installer (using WIX). - Revised and renumbered error codes in C interface.
Renamed HARP_ERROR_SCRIPT and HARP_ERROR_SCRIPT_SYNTAX error codes to
HARP_ERROR_ACTION and HARP_ERROR_ACTION_SYNTAX. Changed HARP_ERROR_PRODUCT
to either HARP_ERROR_INGESTION or HARP_ERROR_IMPORT depending on context.
Added HARP_ERROR_EXPORT, HARP_ERROR_INVALID_PRODUCT and
HARP_ERROR_INVALID_VARIABLE. - The HARP command line tools now link against the dynamic HARP library
instead of the static version. - Updated embedded expat library to version 2.1.1.
- Updated embedded udunits2 library to version 2.2.20.
Also removed the HARP specific udunits XML configuration file since all
HARP units are now included in udunits2. - Renamed all 'stdev' variables to 'uncertainty'.
- Fixed parsing of dimension names; string dimension could be interpreted as
independent dimension. - Improved the efficiency of the conversion from volume mixing ratio to number
density and vice versa. - Update of S5P L2 ingestion module to the lastest version of the product
format specification (S5P-KNMI-L2-0009-SD and relevant DLR PUMs). - Update of S5P L1B ingestion module to the lastest version of the product
format specification (S5P-KNMI-L01B-0012-SD). - Added ingestion modules for the following product types:
- Ozone CCI L2 limb profile
- Ozone CCI L3 limb profile
- Ozone CCI L2 nadir profile
- Ozone CCI L3 nadir profile
- Ozone CCI L4 nadir profile
- Ozone CCI L2 total column
- Ozone CCI L3 total column
- Added public functions harp_get_fill_value_for_type(),
harp_get_valid_min_for_type(), harp_get_valid_max_for_type(),
harp_is_fill_value_for_type(), harp_is_valid_min_for_type(),
harp_is_valid_max_for_type(), harp_ingest_test(),
harp_doc_list_conversions(), harp_doc_export_ingestion_definitions(),
harp_geometry_get_point_distance(), harp_geometry_has_point_in_area(),
harp_geometry_has_area_overlap(), harp_basename(),
harp_product_execute_actions(), harp_variable_set_description(),
harp_variable_set_unit(), harp_variable_set_string_data_element(),
harp_variable_verify(), harp_product_set_source_product(),
harp_product_set_history(), harp_product_verify() - Added public functions for dealing with HARP collocations.
- Moved defines for default harp units to harp.h.
- Added operators =& and !& to the product actions to test integer variables
against a bit mask, where a =& mask is equivalent to (a & mask) != 0 and
a !& mask is equivalent to (a & mask) == 0. - Added functions to verify the internal consistency as well as compliance to
HARP conventions of harp_product and harp_variable instances. Any instance
of harp_product returned by harp_import() or harp_ingest() has been verified
using these functions. - Added derived variable conversion from 'latitude {latitude}' (mid-points) to
'latitude_bounds {latitude, independent}' (bounds); similar for longitude. - Updated HDF5 data format such that HDF5 products written by HARP are
compatible with netCDF-4. HDF5 dimension scales are used to represent shared
dimensions. The "dims" dataset attribute has been deprecated. - Changed netCDF-3/HDF4/HDF5 backends to be more explicit about implicitly
created dimensions (which are used to store scalars and strings for formats
that do not support this natively). - Fixed handling of scalars and empty strings in netCDF-3/HDF4/HDF5 backends.
- Fixed unit attribute name in HARP products ('unit' -> 'units').
- Changed all constants to be in line with CODATA 2014 as published by NIST.
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer almost 9 years ago

harp -
📅 2015-11-09
- First beta release (using CODA 2.15).
Sustainable Development - Environmental Satellites
- C
Published by svniemeijer over 9 years ago
