Data

Tools

Indexes

Applications

Experiments

Open Sustainable Technology

A curated list of open technology projects to sustain a stable climate, energy supply, biodiversity and natural resources.

Recent Releases of Echopype

Echopype - v0.10.1

Overview

This release includes changes that improve performance of compute_Sv and compute_MVBS and fixes of a typo and a bug.

Enhancements

Bug fixes

Documentation

New Contributors

Full Changelog: https://github.com/OSOceanAcoustics/echopype/compare/v0.10.0...v0.10.1

Biosphere - Marine Life and Fishery - Python
Published by leewujung 23 days ago

Echopype - v0.10.0

Overview

This release includes critical updates to migration to support python 3.12, use the new xr.DataTree functionality, and many other changes to resolve deprecation warnings and small bugs. There are a number of pinned dependencies we are continuing to work on.

Enhancements

Bug and warning fixes

Infrastructure

Others

Full Changelog: https://github.com/OSOceanAcoustics/echopype/compare/v0.9.1...v0.10.0

Biosphere - Marine Life and Fishery - Python
Published by leewujung 2 months ago

Echopype - v0.9.1

Overview

This release includes a number of small fixes and enhancements. An important component is to set the max supported python version to <3.12, before we fully migrate to using the new xarray.DataTree in the next release with breaking changes.

Enhancements

Bug and warning fixes

Documentation

Others

New Contributors

Full Changelog: https://github.com/OSOceanAcoustics/echopype/compare/v0.9.0...v0.9.1

Biosphere - Marine Life and Fishery - Python
Published by leewujung 4 months ago

Echopype - v0.9.0

Overview

This release includes a number of new features and performance enhancements, as well as fixes for bugs and warnings. We have fully switched to using the trunk-based development workflow in this release.

New features

Enhancements

Bug and warning fixes

Documentation

Others

New Contributors

Full Changelog: https://github.com/OSOceanAcoustics/echopype/compare/v0.8.4...v0.9.0

Biosphere - Marine Life and Fishery - Python
Published by leewujung 9 months ago

Echopype - v0.8.4

Overview

This release includes a few performance enhancements, bug fixes, and improving warnings/error messages. We are also preparing to switch to using the trunk-based development workflow in the next release to streamline new contributions.

Enhancements

Bug fixes and warnings

Others

New Contributors

Full Changelog: https://github.com/OSOceanAcoustics/echopype/compare/v0.8.3...v0.8.4

Biosphere - Marine Life and Fishery - Python
Published by leewujung about 1 year ago

Echopype - v0.8.3

Overview

This release includes a bug fix for changes from the previous release and a few functionality enhancements.

Enhancements

  • Add parser support for EK80 MRU1 datagram (#1242)
  • Add support for consolidate subpackage functions to accept both in-memory or stored datasets (#1216)
  • Add test for ES60 spare field decoding issue (#1233)
  • Add test for EK80 missing receiver_sampling_freq error (#1234)

Bug fixes

  • Fixed reshape bug in pad_shorter_ping that was remnant from use_swap full refactoring (#1234)

Biosphere - Marine Life and Fishery - Python
Published by leewujung over 1 year ago

Echopype - v0.8.2

Overview

This release includes a number of important performance enhancements, bug fixes, and under-the-hood refactoring to improve code readability.

Enhancements

  • Full refactor of the functionality to directly storing parsed data to zarr before set_groups_* (#1185, #1218, #1070)
    • This avoids memory expansion during NaN padding across channel
  • Improve compute_MVBS using flox (#1124)
  • Revive and improve compute_NASC using flox (#1167)
  • Refactor AZFP XML parser
    • Improved XML parsing by using xml.etree.ElementTree to be consistent with other XML parsers in echopype (#1135)
    • Parse more parameters and store in the resulting EchoData objects (#1135)
    • Add partial support for AZFP multiple phase settings (#1182)
  • Overhaul AZFP Environment group
    • Set mandatory variables not in data to NaN (sounds_speed_indicative, absorption_indicative) (#1226)
    • Parse and store AZFP pressure data when exist (#1189, #1226)
  • Add utils.misc.depth_from_pressure function for converting pressure to depth (#1207)
  • Remove unused mechanism to selectively parse some EK datagrams (#1214)
  • Enhancements of EchoData.update_platform method
    • Factor out utility functions to a submodule (#1209)
    • Auto-assign timestamp for fixed-location external data using first ping_time(#1196)
  • Add support for consolidate subpackage functions to accept both in-memory or stored datasets (#1216) [NOT MERGED]

Bug fixes

  • Fix EchoData.to_zarr encoding (#1128)
    • Eliminate preferred chunks
    • Ensuring chunk alignment and encoding only when handling dask arrays
  • Fix scaling bugs in compute_NASC (#1167)
  • Fix UnicodeDecodeError for ES60 files (#1215)
  • Handle missing receiver_sampling_frequency for EK80 data (#1219)

Infrastructure and refactoring

  • Add support for running individual test files (#1166)
  • Add module based testing (#1180)
  • Update CI to barebone python, without conda (#1192)

Biosphere - Marine Life and Fishery - Python
Published by leewujung over 1 year ago

Echopype - v0.8.1

Overview

This release includes a bug fix and a few infrastructure upgrades and enhancements.

Enhancements and bug fix

  • More intuitive input syntax for frequency_differencing (#1106)
  • Set EK60/80 Platform and NMEA nan timestamp to first ping_time value (#1154)
    • This avoids warnings due to xarray's new behavior with nan timestamp in coordinate
  • Remove automatic version conversion in open_converted (#1143)

Infrastructure

  • Bump testing minimum Python version to 3.9 (#1146)
  • Experiment with Python 3.11 in testing (#1148, #1156)
    • Some tests related to parsed2zarr fail due to the pinned more_itertools version in python 3.11. parsed2zarr is being overhauled in #1070 so these are deferred to the next release.
  • Remove logging during testing (#1141)
  • Remove NASC import causing failure (#1145)

Biosphere - Marine Life and Fishery - Python
Published by leewujung over 1 year ago

Echopype - v0.8.0

Overview

This release includes important updates to the raw-converted data format based on adaptation of convention, an efficiency boost for combining multiple EchoData objects, packaging and infrastructure upgrades, and other updates necessitated by these changes.

Enhancement

  • Overhaul the combine_echodata function (#1042, #1116)
    • No longer require spinning up dask client under the hood during combine
    • Use xr.concat directly compatible with delayed computation
    • No longer require destination path as input argument
  • Overhaul EchoData.update_platform (#1060)
    • Allow adding external variables with arbitrary number of time dimensions, with new time dimensions following an increasing order (e.g., time3, time4, etc)
    • Only allow updating standardized, sonar model-dependent Platform group variables

Raw-converted data format changes

  • Drop the beam and ping_time dimensions for some parameters (#1056, #1083)
    • The dimensions of these parameters were previously expanded to conform with convention requirements, but this inflated the data volume, especially for in memory operations, and made the calibration code confusing
  • Standardize backscatter_r/i long_name in the Sonar/Beam_groupX group, and correct units (#1047)
  • Standardize use of transmit_frequency_start/stop for all echosounder models (#1091)
  • Move filter coefficients and decimation factor to variables in EK80 Vendor_specific group (#1044, #1046, #1105)
  • Add new Provenance group combination_* attributes to combined EchoData object, mirroring the conversion_* attributes (#1113)
  • Bring more consistency in the Platform group across sensors on conversion (#1058, #1061)
  • Add missing mandatory variables and other data format changes (#1094, #1099, #1101, #1102, #1103, #1104, #1107, #1114)
  • Set time arrays explicitly to datetime64[ns] in parsers and remove duplicated timestamp stuff (#1117)

Packaging and infrastructure

  • Remove Pandas<2 pinning (#1080)
  • Pin netcdf to >1.6 and add explicit encoding (#1112)
  • Update CI to prepare for python 3.11 (#1108)
  • Switch github action for get-changed-files (#1118)
  • Fix discrepancy on test log output (#1119)

Others

  • Set and fix encoding for variable encoding (#1072)
    • Specify handling for expected string type
    • Sanitize and set encoding as part of EchoData
  • Handling problematic NMEA messages when setting lat/lon in Platform group (#1067)
  • Update permission check filename to be UUID to make it thread safe (#1110)
  • use pulse_form for EK80 transmit type checks (#1091)
  • Factor out a normalization factor from pulse compression function for flexible use (#1105)
  • Add existence checking for angle parameters in consolidate.add_splitbeam_angle (#1105)
  • Revise qc function to clean up reversed time (#1065)
  • Update matplotlib call to register EK500 colormap (#1068)
  • Temporarily remove compute_NASC for debugging and overhaul (#1136)
  • Remove the deprecated preprocess subpackage (#1077, #1082)
  • Remove the ZarrCombine module since it is no longer used (#1117)
  • Remove outdated ui_param (#1071)

Documentation

  • Add rendering for example EchoData objects for all supported echosounders to show the updated format (#1089)
  • Restructure TOC to have separate Raw data formats and Data Processing sections, each with new subpages (#1093, #1132, #1134)
  • Add discussion on z-axis vertical coordinate system and variables (#1133)
  • Add bibliography page linked from inline citations (#1137)

Biosphere - Marine Life and Fishery - Python
Published by leewujung over 1 year ago

Echopype - v0.7.1

v0.7.1 (2023 May 1)

Overview

This release includes important updates to the requirements, small bug fixes, and refactoring of AZFP parser code to handle files from glider deployment and improve code coherence.

Enhancement and bug fixes

  • AZFP conversion enhancements
    • Add error catch to handle AZFP files with no temperature or tilt, such as those from glider deployments (#1020)
    • Refactor AZFP parser code to improve code coherence (#1024)
    • Correct and add missing SONAR-netCDF4 variable attributes for AZFP (#1023)
  • Improve impedance parameter handling for EK80 files (#1019)
    • allow impedance data type to be float
    • unify naming of impedance parameters
  • Allow only one chirp generation option for EK80 broadband processing (#1019)
    • previously there was an option to use the implementation from Matlab echolab
    • starting from this version only the implementation identical to those from Lars Andersen (repo)
  • Remove unneeded return self in processing level decorator wrapper-inner function for class method (#1037)

Tests and infrastructure

  • Unpin xarray version and pin pandas < 2 (#981)
  • allow v0.7.x in map_ep_version checks

Biosphere - Marine Life and Fishery - Python
Published by leewujung almost 2 years ago

Echopype - v0.7.0

Overview

This release includes new features to interface with Echoview ECS files for computing Sv, reorganization of computing functions into new subpackages, addition of data processing level attributes to data products, and other improvements and bug fixes.

New features and major changes

  • Allow using ECS for calibrating Simrad echosounders (#996, #1004)
    • This functionality is in a beta testing stage
    • Details of implementation may change and bugs are possible
    • Expand ECS parser to accept frequency-dependent values in EK80 ECS files
  • Overhaul env_params to ensure correct intake for calibration (#985)
    • Now allows using env_params entries that are xr.DataArrays
  • Move functions previously in preprocess subpackage to new subpackages (#993). Calling these functions from preprocess is deprecated and will be removed in v0.7.1.
    • clean: remove_noise, estimate_noise
    • commongrid: compute_MVBS, compute_MVBS_index_binning
  • Add commongrid.compute_NASC (#1005)
    • The current implementation uses brute force looping for mean Sv computation, this will be refactored and optimized together with other functions requiring the same pattern in an upcoming release
  • Add global attributes for data processing levels (#1001).
  • Expand mask.apply_mask to handle multi-channel Sv datasets (#1010)
  • Standardize sonar metadata for EK80 data (#992)
    • sonar_serial_number is now an empty global attribute, no longer a variable, as in the EK60 case
    • transducer_name, transducer_serial_number, transceiver_serial_number based on parser parameter transducer_name, transducer_serial_number, and serial_number, respectively

Bug fixes

  • Fix scaling bug for beamwidth_alongship and beamwidth_athwartship from CW-based parameters to values corresponding to center frequency of broadband transmit signals (#998)

Tests and infrastructure

  • Add more comprehensive tests for add_location (#1000)
  • Add test for splitbeam angle ek80_CW_power case (#994)
  • Add unit and integration tests for env_params intake for calibration (#985)
  • Exclude test_data folder in codespell pre-commit hook (#1016)

Biosphere - Marine Life and Fishery - Python
Published by leewujung about 2 years ago

Echopype - v0.6.4.1

Overview

This is a minor release that includes a small bug fix and an enhancement, which removes excessive warnings due to changes in v0.6.4.

Bug fix

  • Fix a bug that prevented passing in env_params for EK60 calibration (#987)

Enhancement

  • Handling non-positive values that causes log10 warnings on EK calibration (#986)

Biosphere - Marine Life and Fishery - Python
Published by leewujung about 2 years ago

Echopype - v0.6.4

Overview

This is a release that includes important performance enhancements that allow user-provided calibration parameters for broadband and narrowband calibration, new functionalities to compute and attach split-beam angles to calibrated Sv dataset, perform frequency-differencing and masking, as well as a number of bug fixes and other improvements.

New features

  • Allow passing in cal_params as a dictionary for narrowband and broadband calibration (#955)
  • Add default chunk encoding for zarr output (#939)
  • Add add_splitbeam_angle function to the consolidate subpackage (#916, #971)
  • Add a new mask subpackage
    • Add apply_mask function to the mask subpackage (#912)
    • Add frequency-differencing function to the mask subpackage (#901)
  • Allow selection of a subset of channels when combining multiple echodata objects (#892)
    • This is done via the added channel_selection input argument to combine_echodata
  • Add default consolidated flag for echodata.to_zarr (#855)
  • Generalize and improve efficiency of compute_MVBS (#878)
    • Allow echo_range that vary with ping_time
    • Allow Sv data that are dask arrays
  • Add .nbytes to obtain that data size of an echodata object (#874)
  • Add new data variables from the raw data files to facilitate EK80 calibration (#944)
    • transceiver type, transmit impedance, receive impedance, receiver sampling frequency

Enhancements and other changes

  • Unify the order of dimensions of echo_range across all sonar models (#968)
  • Create and use the default echopype home directory (#896, #954)
    • This default directory is at ~/.echopype
  • Refactor the calibrate subpackage (#904)
    • routines for env_params and cal_params intake and range computation are now in different modules rather than methods in each classes
    • Tidy up cal_params related routines (#953)
    • Revise env_params related routines (#952)
  • Improving pulse compressed broadband Sv computation and echo range computation (#944)
    • The resulting values are tested against pyEcholab and Echoview outputs
    • Note there is unresolved discrepancy between pyEcholab/Echopype outputs with Echoview outputs for the first section of Sv values
  • Change input argument names for open_raw (#962)
    • Change offload_to_zarr to use_swap
    • Change max_zarr_mb to max_mb

Bug fixes

  • Handling of provenance attributes in apply-mask and add-depth, especially for testing (#930)
  • Fix bugs in combining multiple echodata objects (the combine_echodata function)
    • Fix meta_source_filenames bug and enable (meta)source_filenames appending of path and list (#913)
  • Fix bugs in env_params intake for calibration (#952)

Infrastructure

  • Update docker build for arm64 mac silicon chips (#964)
  • Fix windows github actions workflow (#947)
  • Remove mamba dependency (#946)
  • Fix pre-commit ci and update RTD config for jupyter-book (#934)

Biosphere - Marine Life and Fishery - Python
Published by leewujung about 2 years ago

Echopype - v0.6.3

Overview

This is a minor release that includes an important performance enhancement for combining large volumes of data residing in individual files into a single entity, a number of bug fixes, and other smaller improvements.

New features

  • Overhaul combine_echodata function
    • Allow combine a large number of EchoData objects exceeding memory limits (#808, #824, #830)
    • Remove reversed time check from combine_echodata (#835)
    • Add minimal ZarrCombine test (#826)
    • Order the channel coordinate in file conversion to ensure consistent combination across files (#818)
    • Revise outdated data combination behavior (#797, #799)
    • Clean up all coordinate and attribute details under combine_echodata function (#848, #849)
  • Track provanance for filenames of raw data files and auxiliary files
    • Propagate xml_path as meta_source_filenames to combined echodata (#814, #852)
    • Standardize handling of source_filenames in individual and combined echodata as well as downstream datasets (#804, #806)

Under the hood enhancements

  • Clean up functions for setting encoding in the converted files (#851)
  • Add requests and aiohttp to dependency (#844)
  • Pin netcdf4 to be <1.6 for pypi package due to ongoing netcdf-c problem (#843)
  • Write Parsed2Zarr generated files to temp_echopype_output/parsed2zarr_temp_files (#832)
  • Change isel to sel to fix compute_Sv to allow working with dask array (#828)
  • Add open_raw(offload_to_zarr=True) integration tests (#794)

Bug fixes

  • Fix regression bug with interpolating environmental variables to ping_time grid (#837, #856)
  • Fix WindowsPath error with compute_Sv when run on Windows (#829)
  • Fix logic problem in open_raw(offload_to_zarr=True) that sometimes cause problems (#794, #853)

Infrastructure

  • Update CI to use micromamba (#805)
  • Fix version string on CI (#804, #820)
  • Rename ci.yaml to build.yaml for clarity (#807)

Biosphere - Marine Life and Fishery - Python
Published by leewujung over 2 years ago

Echopype - v0.6.2

Overview

This is a minor release that includes a few new features and memory efficiency-related changes that make echopype better.

New features

  • Add a new subpackage consolidate that contains functions to consolidate data across the calibrated Sv dataset and the corresponding raw-converted file
  • Add function consolidate.add_location to interpolate location to calibrated dataset (#749)
  • Add function consolidate.add_depth to convert range_meter to depth with information on transducer tilt and depth (#738)
  • Move function that swaps the channel with frequency dimension (now as consolidate.swap_dims_channel_frequency) (#738)
  • Add new functionality to allow control of logging outputs (#772)

Under the hood enhancements

  • Improve memory usage while converting files that require significant NaN-padding and previously would incur very large memory expansion (#774)
    • This is achieved by directly writing variables that may incur a large memory expansion into a temporary zarr store
    • Beta function that will benefit from user feedback
  • Overhaul access pattern for EchoData (#762)
    • Remove previous access pattern for different groups in the raw-converted file
    • Starting from this release all groups are accessed with echodata["GROUP_PATH"], e.g., echodata["Platform"], echodata["Sonar/Beam_group1"], etc.
  • Make long_name in ds_power for EK80 consistent with other sonar model (#771)
  • Modify set_beam() so it returns a list (#780)
  • Change the order in _save_groups_to_file so the order of groups is preserved when opening a converted netcdf file (#779)
  • Remove the user option to select NMEA sentences in open_raw to ensure raw data is preserved (#778)

Infrastructure

  • Update CI set up to avoid exceeding GitHub actions memory limitation (#761)
    • decrease the number of workers from 4 to 2

Biosphere - Marine Life and Fishery - Python
Published by leewujung over 2 years ago

Echopype - v0.6.1

Overview

This is a minor release that includes important bug fixes, a number of new features, and some leftover data format changes specific to parsed AD2CP data format.

Bug fixes

  • We use datatree under the hood for the EchoData object, but datatree v0.0.4 had a bug in accessing subgroups in netCDF data model in Windows OS. This was fixed in datatree v0.0.6, and hence we updated our dependency and made other associated changes to support cross platform users (#732, #748)
  • Fix empty Sv problem related to renaming time coordinate associated with environmental parameters used for calibration; this is a regression bug from v0.6.0 changes (#755)
  • Fix a bug in compute_MVBS in selecting echo_range for specific frequency. This is a regression bug from v0.6.0 changes (#736)
  • Allow parsing data from EK60 with split-beam transducers but without phase/angle data (#491, #718)
  • Check filter coeffs existence in SetGroupsEK80.set_vendor before saving (#720, #724)
  • Fix the check in compute_MVBS for handling different variations of NaN entries in the Sv dataset (#753)
  • Fix invalid timestamp issue in AD2CP data conversion (#733)

New features

  • Enhance update_platform to support a new use case (location data from fixed location) and add more consistency (#741)
  • Ability to parse and store RAW4 datagram for EK80 data (#714)
  • Add utility function for swapping channel coordinate with frequency_nominal (#710)
  • Add ES70, ES80, EA640 to allowed data type for calibration (#759)

Changes of netCDF data model

  • Reorganize AD2CP data variables into different Sonar/Beam_groupXs and different first-level groups in a form consistent with v0.6.0 changes for all other sonar models (#731); some variables remain to be discussed and may change in future releases (#719)

Enhancements

  • Refactor AD2CP conversion to improve speed and memory usage through removal of xr.merge (#505)
  • Update Python requirements in docs to >=3.8 (#744)

Infrastructure

  • Update PR action to use PR title [all tests ci] to run the entire suite of tests and [skip ci] to skip all tests (#721)

Biosphere - Marine Life and Fishery - Python
Published by leewujung almost 3 years ago

Echopype - v0.6.0

Overview

This is a major release that contains changes that enhances the compliance of echopype data model (and hence generated file structure) to the SONAR-netCDF4 convention Version 1.0 . In addition, some variables were renamed to improve intuitive understanding of sonar data, provenance and standardized attributes are added to the processed dataset (e.g. Sv), the deprecated old API (<0.5.0) was removed, and some bugs were fixed.

Changes of netCDF data model

  • Move and rename the original Beam and Beam_power group to be subgroups under the Sonar group, in order to comply with the structure defined in the convention (#567, #574, #605, #606, #611)
    • Beam --> Sonar/Beam_group1: contains either raw power or power/angle data for all sonar models other than EK80. For EK80, if only complex or power/angle data exist, all data are in this group; if both complex and power/angle data exist, the complex data are in this group.
    • Beam_power --> Sonar/Beam_group2: contains power/angle when complex data occupies Sonar/Beam_group1; only exists for EK80 data when both power/angle data and complex data bothexist in the file
  • Rename the coordinate range_bin to range_sample to make it obvious that this coordinate indicates the digitization sample number for the sound waveform or intensity time series, and hence it takes a form of sequential integers 0, 1, 2, 3, ... (#595)
  • Rename the data variable range in the calibrated Sv or TS dataset to echo_range, so that it is not confused with the python built-in function (#590)
  • Rename the coordinate quadrant for EK80 data to beam (#619)
  • Add coordinate beam with length 1 for all sonar models, except for AD2CP (#638, #646)
  • Rename the data variable Sp to TS since "point backscattering strength" is a Simrad terminology and target strength (TS) is clearly defined and widely used. (#615)
  • Rename time dimensions in the Platform group (location_time: time1, mru_time: time2) (#518, #631, #647)
  • Rename the coordinate frequency to channel for all groups, to be more flexible (can accommodate channels with identical frequencies #490) and reasonable (since for broadband data the channel frequency is only nominal #566) (#657)
  • Rename the data variable heave to vertical_offset in the Platform group (#592, #623)
  • Change src_filenames string attribute to source_filenames list-of-strings variable (#620, #621)
  • Bring consistency to the names of the time coordinates for the Platform and Environment groups (#656, #672)

Changes of EchoData group access pattern

  • The groups can now be accessed via a path in the form echodata["Sonar/Platform"], echodata["Sonar/Platform"], echodata["Sonar/Beam_groupX"], etc. using DataTree v0.0.4 functionalities. (#611)
  • The previous access pattern echodata.platform, echodata.sonar, echodata.beam, etc. is deprecated and will be removed in v0.6.1.

Addition of attributes and variables in raw-converted and processed data

  • Add indexing info for Beam_groupX as data variable under the Sonar group (#658)
  • Add missing coordinate and variable attributes in the processed datasets Sv, MVBS, TS(#594)
  • Add water_level to processed datasets (Sv, MVBS, TS) for convenient downstream conversion to depth (#259, #583, #615)
  • Add additional environment variables for EK80 data (#616)
  • Add missing platform data variables (#592, #631, #649, #654)

New features and other enhancements

  • Add parser for Echoview ECS file (#510)
  • Add provenance to raw-converted and processed datasets (#621)
  • Consolidate convention specs into a single yml file for pre-loading when creating EchoData objects (#565)
  • Extra visualization module can now handle both frequency and channel filtering, as well as files with duplicated frequencies (#660)
  • Improve selection of Vendor-specific calibration parameters for narrowband EK data (#697)

CI improvements

  • Upgrade python minimum to 3.8 and tidy up packaging (#604, #608, #609)
  • Upgrade echopype development status to Beta (#610)
  • Update setup-services.py to include images & volumes subtleties (#651)

Other changes

  • Remove the deprecated old (<0.5.0) API (#506, #601)
  • Update README in the echopype/test_data folder (#584)
  • Add documentation for visualization (#655)
  • Add development roadmap to documentation (#636, #688)
  • Restructure and expand data format section (#635)

Biosphere - Marine Life and Fishery - Python
Published by leewujung almost 3 years ago

Echopype - v0.5.6

This is a minor release that contains an experimental new feature and a number of enhancements, clean-up and bug fixes, which pave the way for the next major release.

New feature

  • (beta) Allow interpolating CTD data in calibration (#464)
    • Interpolation currently allowed along the ping_time dimension (the "stationary" case) and across latitude and longitude (the "mobile" case).
    • This mechanism is enabled via a new EnvParams class at input of calibration functions.

Enhancements

  • Make visualize module fully optional with matplotlib, cmocean being optional dependency (#526, #559)
  • Set range entries with no backscatter data to NaN in output of echodata.compute_range() (#547) and still allows quick visualization (#555)
  • Add codespell GitHub action to ensure correct spellings of words (#557)
  • Allow sonar_model="EA640" for open_raw (before it had to be "EK80") (#539)

Bug fixes

  • Allow using sonar_model="EA640" (#538, #539)
  • Allow flexible and empty environment variables in EA640/EK80 files (#537)
  • Docstring overhaul and fix bugs in utils.uwa (#525)

Documentation

  • Upgrade echopype docs to use jupyter book (#543)
  • Change the RTD latest to point to the dev branch (#467)

Testing

  • Update convert tests to enable parallel testing (#556)
  • Overhaul tests (#523, #498)
    • use pytest.fixture for testing
    • add ES70/ES80/EA640 test files
    • add new EK80 small test files with parameter combinations
    • reduce size for a subset of large EK80 test data files
  • Add packaging testing for the dev branch (#554)

Biosphere - Marine Life and Fishery - Python
Published by leewujung about 3 years ago

Echopype - v0.5.5

This is a minor release that includes new features, enhancements, bug fixes, and linking to an echopype preprint.

New features

  • Allow converting ES60/70/80 files and handle various datagram anomaly (#409)
  • Add simple echogram plotting functionality (beta) (#436)

Enhancements

  • update_platform method for EchoData now include proper variable attributes and correctly selects time range of platform data variables corresponding to those of the acoustic data (#476, #492, #493, #488)
  • Improve testing for preprocess.compute_MVBS by running through real data for all supported sonar models (#454)
  • Generalize handling of Beam group coordinate attributes and a subset of variable attributes (#480, #493)
  • Allow optional kwargs when loading EchoData groups to enable delaying operations (#456)

Bug fixes

  • The gain factor for band-integrated Sv is now computed from broadband calibration data stored in the Vendor group (when available) or use nominal narrowband values (#446, #477)
  • Fix time variable encoding for combine_echodata (#486)
  • Fix missing ping_time dimension in AZFP Sv dataset to enable MVBS computation (#453)
  • Fix bugs re path when writing to cloud (#462)

Documentation

  • Improvements to the "Contributing to echopype" page: Elaborate on the git branch workflow. Add description of PR squash and merge vs merge commit. Add instructions for running only a subset of tests locally (#482)
  • Add documentation about output_storage_options for writing to cloud storage (#482)
  • Add documentation and docstring for sonar_model in open_raw (#475)
  • Improve documentation of EchoData object by adding a sample of the xarray Dataset HTML browser (#503)

Others

Biosphere - Marine Life and Fishery - Python
Published by leewujung over 3 years ago

Echopype - v0.5.4

Overview

This is a minor release that contains a few bug fixes and new functionalities.
The repo has migrated to use main instead of master after this release.

New features

  • Adding external platform-related data (e.g., latitude, longitude) to the EchoData object via the update_platform method (#434)
  • Allow converting and storing data with duplicated ping times (#433)
  • Add simple functions to compute summary statistics under the metrics subpackage (#444)

Bug fixes

  • Allow string info in AD2CP data packet header (#438)
  • Re-attach sonar_model attribute to outputs of combine_echodata (#437)
  • Handle exception in open_converted due to potentially empty beam_power group in Zarr files (#447)

Others

  • Warn users of removal of old API in the next release (#443)

Biosphere - Marine Life and Fishery - Python
Published by leewujung over 3 years ago

Echopype - v0.5.3

Overview

This is a minor release that adds a few new functionalities, in particular a method to combine multiple EchoData objects, addresses a few bugs, improves packaging by removing pinning for dependencies, and improving the testing framework.

New features

  • Add a new method to combine multiple EchoData objects (#383, #414, #422, #425 )
    • Potential time reversal problems in time coordinates (e.g., ping_time, location_time) are checked and corrected as part of the combine function
    • The original timestamps are stored in the Provenance group
  • Add a new method compute_range for EchoData object (#400)
  • Allow flexible extensions for AZFP files in the form ".XXY" where XX is a number and Y is a letter (#428)

Bug fixes

  • Fix the bug/logic problems that prevented calibrating data in EK80 files that contains coexisting BB and CW data (#400)
  • Fix the bug that prevented using the latest version of fsspec (#401)
  • Fix the bug that placed echosounder_raw_transmit_samples_i/q as the first ping in echosounder_raw_samples_i/q as they should be separate variables (#427)

Improvements

  • Consolidate functions that handle local/remote paths and checking file existence (#401)
  • Unpin all dependencies (#401)
  • Improve test coverage accuracy (#411)
  • Improve testing structure to match with subpackage structure (#401, #416, #429 )

Documentaion

  • Expand Contributing to echopype page, including development workflow and testing strategy (#417, #420, #423)

Biosphere - Marine Life and Fishery - Python
Published by leewujung over 3 years ago

Echopype - v0.5.2

Overview

This is a minor release that addresses issues related to time encoding for data variables related to platform locations and data conversion/encoding for AD2CP data files.

Bug fixes and improvements

  • Fixed the location_time encoding in the Platform group for latitude and longitude data variables (#393)
  • Fixed the location_time encoding in the Platform/NMEA group (#395)
  • Updated EchoData repr to show Platform/NMEA (#395, #396)
  • Improved AD2CP data parsing and conversion (#388)
    • Cleaned up organization of data from different sampling modes and their corresponding time coordinates
    • Fixed parsing issues that generated spikes in parsed echosounder mode amplitude data
    • Removed the Beam_complex group and put raw IQ samples in the Vendor group per convention requirements
    • Populated the Sonar group with AD2CP information

Biosphere - Marine Life and Fishery - Python
Published by leewujung almost 4 years ago

Echopype - v0.5.1

Overview

This is a minor release that addresses a couple of issues from the last major version (0.5.0) and improves code maintenance and testing procedures. See v0.5.0 release notes for important API changes relative to previous versions.

New features

  • Added experimental functions to detect and correct ping_time reversals.
    See qc subpackage (#297)

Updates and bug fixes

  • Fixed ADCP encoding issues (#361)
  • Updated SetGroupsBase to use ABC (Abstract Base Classes) Interface (#366)
  • Whole code-base linted for pep8 (#317)
  • Removed old test data from the repository, including from Git LFS (#369, #271)
  • Updated package dependencies (#365)
  • Simplified requirements for setting up local test environment (#375)

CI improvements

  • Added code coverage checking (#317)
  • Added version check for echopype install (#367, #370)

Biosphere - Marine Life and Fishery - Python
Published by lsetiawan almost 4 years ago

Echopype - v0.5.0 (Major API update)

Overview

This release includes:

  • major API updates to provide a more coherent data access pattern
  • restructuring of subpackages and classes to allow better maintenance and future expansion
  • reorganization of documentation, which also documents the API changes
  • overhaul and improvements of CI, including removing the use of Git LFS to store test data
  • new features
  • bug fixes

API updates

The existing API for converting files from raw instrument formats to a standardized format, and for calibrating data and performing operations such as binned averages and noise removal has been updated.

The new API uses a new EchoData object to encapsulate all data and metadata related to/parsed from a raw instrument data file. Beyond the calibration of backscatter quantities, other processing functions follow a consistent form to take an xarray Dataset as input argument and returns another xarray Dataset as output.

The major changes include:

  • change from an object-oriented method calls to functional calls for file conversion (using the new convert subpackage), and deprecate the previous Convert class for handling file parsing and conversion
  • deprecate the previous Process class, which use object-oriented method calls for performing both calibration and data processing
  • separate out calibration functions to a new calibrate subpackage
  • separate out noise removal and data reduction functions to a new preprocess subpackage
  • create a new EchoData object class that encapsulates all raw data and metadata from instrument data files, regardless of whether the data is being parsed directly from the raw binary instrument files (returned by the new function open_raw) or being read from an already converted file (returned by the new function open_converted)

See documentation for detailed usage of the new API.

Subpackage and class restructuring

The subpackages and classes were restructured to improve modularity that will help will future expansion and maintenance. The major restructuring includes:
("SONAR" below is used to indicate the sonar model, such as EK60, EK80 or AZFP)

  • consolidate overlapping EK60/EK80 components, deprecate the previous Convert classes that handled file parsing and serialization, and revise new ParseSONAR and SetGroupsSONAR classes for file parsing and serialization
  • consolidate all calibration-related components to a new calibrate submodule, which uses CalibrateSONAR classes under the hood
  • consolidate all preprocessing functions into a a new preprocess submodule, which will be later expanded to include other functions with similar use in a workflow

CI overhaul and improvements

  • Added github workflows for testing, building test docker images, and publishing directly to PyPI
  • Deprecated usage of Travis CI
  • Test run is now selective on Github, to run tests only on changed/added files. Or run all locally with run-test.py script. (#280, #302)

Documentation reorganization and updates

  • Re-organization of pages with better grouping
  • Added What's New page
  • Added Contributing to echopype page
  • Overhaul API reference page

New features

  • Add interfacing capability to read from and write to cloud object storage directly. (#216, #240)
  • Add support to convert .ad2cp files generated by Nortek's Signature series ADCP (#326)
  • Allow environmental and calibration parameters to be optionally used in calibration in place of the values stored in data file
  • Mean volume backscattering strength (MVBS) can now be computed based on actual time interval (specified in seconds) and range (specified in meters) (#54)
  • Add NMEA message type as a data variable in the Platform group (#232), which allows users to freely select the suitable ones depending on use

Bug fixes

  • Fix EK80 config XML parsing problem for files containing either PulseDuration or PulseLength (#305)
  • Fix time encoding discrepancy in AZFP conversion (#328)
  • Fix problematic automatic encoding of AZFP frequency (previously as int) to float64 (#309)
  • Overhaul EK80 pulse compressed calibration (current implementation remaining in beta, see #308)

Biosphere - Marine Life and Fishery - Python
Published by leewujung almost 4 years ago

Echopype - Patches and enhancements to file conversion

This minor release includes the following changes:

Bug fixes

  • Fix bug in top level .nc output when combining multiple AZPF .01A files
  • Correct time stamp for .raw MRU data to be from the MRU datagram, instead of those from the RAW3 datagrams (although they are identical from the test files we have).
  • Remove unused parameter sa_correction from broadband .raw files
  • Make sure import statement works on Google colab

Enhancements

  • Parse Simrad EK80 config XML correctly for data generated by WBAT and WBT Mini, and those involving the 2-in-1 "combi" transducer
  • Parse Simrad .raw files with NME1 datagram, such as files generated by the Simrad EA640 echosounder
  • Handle missing or partially valid GPS data in .raw files by padding with NaN
  • Handle missing MRU data in .raw files by padding with NaN
  • Parse .raw filename with postfix beyond HHMMSS
  • Allow export EK80 XML configuration datagram as a separate XML file

Notes

To increase maintenance efficiency and code readability we are refactoring the convert and process modules. Some usage of these modules will change in the next major release.

Biosphere - Marine Life and Fishery - Python
Published by leewujung over 4 years ago

Echopype - Add EK80 conversion, rename subpackage model to process

New features

  • Add EK80 support:
    • File conversion from EK80 .raw files to netCDF and zarr formats
    • "Simple" calibration to frequency-average Sv based on pulse compression output is implemented but needs to be thoroughly tested.
  • Rename subpackage echopype.model to echopype.process
    • The new name better describes the subpackage's function to process data for further analysis
    • Also rename class EchoData to Process to mirror the structure in Convert better.
    • Importing using the old names will be deprecated in the next release.
  • Overhaul converting multiple files with combine_opt=True
    • If target format is netCDF, temporary files will be created and finally combined to a single netCDF. This is due to current restriction that xarray does not allow simply appending new data to an existing file.
    • If target format is zarr, data in each file are unpacked and appended to the same output file.
  • Allow reading Zarr into Process in addition to netCDF: thanks @lsetiawan!
  • Add a logo!

Bug fixes

  • Fix bugs in slicing NMEA group data based on the same time base when range_bin is changed

Biosphere - Marine Life and Fishery - Python
Published by leewujung almost 5 years ago

Echopype - Bug fix for remove noise and testing env update

This minor release includes the following:

  • fix the bug in remove_noise function for AZFP data (issue #110)
  • update testing to use both python 3.7 and 3.8
  • include how to pull test files using git LFS in documentation
  • remove a stalled command line file converter that won't be further developed

Biosphere - Marine Life and Fishery - Python
Published by leewujung almost 5 years ago

Echopype - Bug fixes and calibration parameter updates

This minor release includes a few bug fixes:

  • fix zarr saving error in the Provenance group in the previous release
  • fix bugs in updating environmental parameters (used for calibration) for EK60 and AZFP
  • allow users to update calibration parameters in EK60
  • update notebook EK60_demo_OOI.ipynb to use new raw data archive folder structure

Biosphere - Marine Life and Fishery - Python
Published by leewujung about 5 years ago

Echopype - Add more convert options, accommodate range differences

This release contains updates to the convert and model module:

  • Correct file parsing and storage for split-beam angle.
  • Allow user-defined environmental variables to be used in calibration.
  • Accommodate EK60 setting variabilities in the range_bin dimension.
    • The raw file will be split into multiple .nc files if the length of range_bin changes in the middle.
    • For cases when different frequency has different length of range_bin, the shorter channels are padded with NaN.
  • Allow combining multiple files during unpacking.
  • Options to:
    • overwrite previously converted files.
    • specify output directory and filename postfix for .calibration() and .get_MVBS() operations.
  • Add support to convert to .zarr file. Note processing functionalities have not been fully supported.

Biosphere - Marine Life and Fishery - Python
Published by leewujung about 5 years ago

Echopype - Add support for AZFP with uniform interface

This release includes major changes to the package structure and functionality improvements:

  • Added support for handling .01A data files from the AZFP echosounder.
  • Changed the module structure to enable a uniform interface to work with EK60 and AZFP data.
  • Improvements to EK60 components:
    • Included NMEA datagram unpacking in file conversion routine.
    • Added handling for pings with different range setting within the same data file. This feature is experimental and the default behavior will change in future release.

Biosphere - Marine Life and Fishery - Python
Published by leewujung over 5 years ago

Echopype - Updated versioning setup

Minor update:

  • use versioneer for auto-versioning

Biosphere - Marine Life and Fishery - Python
Published by leewujung about 6 years ago

Echopype - Added EK60 data model and methods

Biosphere - Marine Life and Fishery - Python
Published by leewujung about 6 years ago