Releases | Open Sustainable Technology

pydov - v4.0.0

Breaking changes
- Following generalisation and standardisation of DOV object types regarding samples (monsters) and observations,
  there are changes in pydov too. Following datatypes are impacted:
  - Grondmonsters (borehole samples): removed
    
    pydov.search.grondmonster and pydov.types.grondmonster modules have been removed, in favor of the generic Monster type.
    Use pydov.search.monster and pydov.types.monster instead.
    Consult the available fields, fieldsets and subtypes in the datasets section <dataset_monster>
    and check the tutorial <notebooks/search_monsters> for more information and examples.
  - Grondwatermonsters (groundwater samples): removed
    
    pydov.search.grondwatermonster and pydov.types.grondwatermonster modules have been removed, in favor of the generic Monster type.
    Use pydov.search.monster and pydov.types.monster instead.
    Consult the available fields, fieldsets and subtypes in the datasets section <dataset_monster>
    and check the tutorial <notebooks/search_monsters> for more information and examples.
  - Bodemmonsters (soil samples): removed
    
    pydov.search.bodemmonster and pydov.types.bodemmonster modules have been removed, in favor of the generic Monster type.
    Use pydov.search.monster and pydov.types.monster instead.
    Consult the available fields, fieldsets and subtypes in the datasets section <dataset_monster>
    and check the tutorial <notebooks/search_monsters> for more information and examples.
  - Bodemobservaties (soil observations): removed
    
    pydov.search.bodemobservatie and pydov.types.bodemobservatie modules have been removed, in favor of the generic Observatie type.
    Use pydov.search.observatie and pydov.types.observatie instead.
    Consult the available fields, fieldsets and subtypes in the datasets section <dataset_observatie>
    and check the tutorial <notebooks/search_observaties> for more information and examples.
- Preparing the pending deprecation of the Lambert72 coordinate reference system (CRS) (EPSG:31370), all locations and geometry return fields
  now require explicit CRS definition. This impacts:
  - pydov.util.location.Box - add epsg=31370 to specify Lambert72.
  - pydov.util.location.Point - add epsg=31370 to specify Lambert72.
  - pydov.util.location.GmlObject - requires explicit srsName.
  - pydov.search.fields.GeometryReturnField - add epsg=31370 to specify Lambert72.
  For the same reason, it's no longer possible to reference a geometry return field by string reference, you have to use a GeometryReturnField instance
  from now on, explicitly defining the desired CRS.
- Fields with an associated codelist now support both codes, labels and definitions. This means that they are no longer listed as dict in the output of get_fields(),
  but are descedents of pydov.util.codelists.AbstractCodeList. Codelists can originate from either a feature catalogue, an associated XSD type or (new) an OSLO
  codelist. If you want to add either the labels or the definitions to your output dataframes, you can use the get_label and get_definition methods, as
  explained in the documentation <map_codelist>.
- Field classes restructuring:
  - The GeometryReturnField and ReturnField classes now reside inside the pydov.search.fields package instead of pydov.types.fields,
    see pydov.search.fields.GeometryReturnField and pydov.search.fields.ReturnField
- Multivalued fields now use tuples instead of lists. This has the benefit of being hashable, enabling some features in Pandas that are otherwise not available (e.g. drop duplicates).
  This is also reflected in the output of get_fields(), where the property list: True/False has been replaced with multivalue: True/False.
  More information in the documentation <available_attribute_fields>.
New features
- Add support for extra fieldsets
  
  Starting from pydov 4, we support extra fieldsets for certain object types. These are fields that are not available in the the default output dataframes,
  but can easily be added should you need them.
  
  You can explore which fieldsets are available with the get_fieldsets() method on the main type::
  
  from pydov.types.boring import Boring
  print(Boring.get_fieldsets())
  
  You can add the fields from a fieldset to your search using the with_extra_fields() method::
  
  from pydov.search.boring import BoringSearch
  from pydov.types.boring import MethodeXyz
  
  boring_search = BoringSearch(objecttype=Boring.with_extra_fields(MethodeXyz))
  df = boring_search.search(max_features=10)
  print(df)
  
  More information can be found in the customizing output <adding_extra_fields> section.
- Add support for extra subtypes
  
  We now also support multiple subtypes per main type. Most main types by default still have a subtype, but for some we also have other subtypes available.
  
  Check which subtypes are available with the get_subtypes() method on the main type::
  
  from pydov.types.boring import Boring
  print(Boring.get_subtypes())
  
  Use one of the subtypes in your search using the with_subtype() method::
  
  from pydov.search.boring import BoringSearch
  from pydov.types.boring import Kleur
  
  boring_search = BoringSearch(objecttype=Boring.with_subtype(Kleur))
  df = boring_search.search(max_features=10)
  print(df)
  
  More information can be found in the customizing output <switching_subtypes> section.
- Monster (samples): added new type for generic Monster type
  
  Added a new object type for the generic Monster type. Check all available fields, fieldsets and subtypes in
  the datasets section <dataset_monster>. Consult the tutorial <notebooks/search_monsters> for more
  information and examples.
- Observatie (observaties): added new type for generic Observatie type
  
  Added a new object type for the generic Monster type. Check all available fields, fieldsets and subtypes in
  the datasets section <dataset_observatie>. Consult the tutorial <notebooks/search_observaties> for more
  information and examples.
- Added visual output for search instances and get_fields() output in Jupyter notebooks.
  
  Both search instances as well as the output of get_fields() and its subitems now support visualisation in Jupyter notebooks.
  To get a quick overview of a datatype, visualise its search instance::
  
  from pydov.search.boring import BoringSearch
  BoringSearch()
  
  The output of get_fields() now supports both indexed as well as attribute-style access::
  
  from pydov.search.boring import BoringSearch
  boring_search = BoringSearch()
  
  boring_search.get_fields()['methode']['codelist']['spade']
  
  boring_search.get_fields().methode.codelist.spade
  
  This output is also available in the documentation tutorials, e.g. for Boring <notebooks/search_boringen>.
Documentation-only updates
- Updated the documentation regarding object type customization with the newly introduced with_extra_fields() and with_subtype() methods.
  The old syntax should still be supported too. Check the new documentation here <custom_object_types>.

Natural Resources - Soil and Land - Python
Published by Roel 2 months ago

pydov - v3.3.1

Fixes and improvements
- Following WFS/XSD schema updates, update the GrondwaterVergunning and GrondwaterFilter types to support HCOVv2 coding (#428)

Natural Resources - Soil and Land - Python
Published by Roel 8 months ago

pydov - v3.3.0

New features
- Introduce PropertyLikeList and FuzzyJoin query utilities to search for, and join on, fuzzy (non-exact) matches. (#406)
- Add support for splitting field values into a list of values. (#407)
Fixes and improvements
- Update grondmonster type based on WFS/XSD schema updates. (#408)
- Always request geometry using urn:ogc: srsName to fix coordinate ordering. (#413, #418)

Natural Resources - Soil and Land - Python
Published by Roel 11 months ago

pydov - v3.2.0

News
- This version drops support for Python 3.8.
- This version adds support for Python 3.12 and Python 3.13.
- This version is supported on Python 3.9, 3.10, 3.11, 3.12 and 3.13.
Fixes and improvements
- Update grondmonster XML based on XSD schema updates. (#404)
- Fix GMLObject tests. (#401)

Natural Resources - Soil and Land - Python
Published by Roel about 1 year ago

pydov - v3.1.0

News
- Add contribution section in the pydov repository. (#394)
New features
- Add support for generic DOV WFS layers. Next to the existing 'fixed' datatypes, this make all other WFS data we publish available in pydov too, allowing to use them using the same search options as for other datatypes. (#383)
  
  To support this we are now also more relaxed about metadata being available: we will still use it when it's available, but will no longer raise errors when it's not. While all of our layers should have metadata available, most of them don't have a feature catalogue. We will now return 'None' in the description of get_fields() when this is missing.
- Add support for geometry return fields. (#386)
  
  While it was already possible to do advanced spatial queries, up until now the resulting dataframes only contained attribute fields and not the geometry of the feature. When the necessary dependencies are installed, it is now possible to add geometry fields to your dataframe using the return_fields parameter. Converting the resulting dataframe into a GeoPandas geodataframe then becomes easy.
Fixes and improvements
- Generate stable and unique GML ids for Box and Point locations. (#387, #388)
- Fix property name replacement in sort_by. (#390, #391)
- Update mv_mtaw fields based on updated DOV XML schema. (#395)
Documentation-only updates
- Open tutorial notebooks through documentation. (#385)

Natural Resources - Soil and Land - Python
Published by Roel over 2 years ago

pydov - v3.0.0

News
- This version drops support for Python 3.7.
- This version adds support for Python 3.11.
- This version is supported on Python 3.8, 3.9, 3.10 and 3.11.
Breaking changes
- pydov3 uses WFS 2.0.0 instead of WFS 1.1.0, as a consequence attribute filters should now use FES2.0 and location filters should now use GML3.2. This change impacts a number of places:
  - query parameter in the pydov.search.abstract.AbstractSearch.search method
    
    Attribute query operators, like PropertyIsEqualTo, PropertyIsGreaterThan and so on, should from now on be imported from the owslib.fes2 package instead of the owslib.fes package. E.g.:
```
 # change this
 from owslib.fes import And, PropertyIsEqualTo

 # into this
 from owslib.fes2 import And, PropertyIsEqualTo
```
  - sort_by parameter in the pydov.search.abstract.AbstractSearch.search method
    
    Also the SortBy operator should from now on be imported from the the owslib.fes2 package instead of the owslib.fes package. E.g.:
```
  # change this
  from owslib.fes import SortBy, SortProperty

  # into this
  from owslib.fes2 import SortBy, SortProperty
```
  - The pydov.util.location.GmlObject class now expects GML3.2 geometries instead of GML3.1.1.
    
    Use GML3.2 objects from now on.
  - The pydov.util.location.GmlFilter class now expects GML3.2 documents instead of GML3.1.1.
    
    Transform the document to GML3.2 yourself or use a pydov.util.location.GeometryFilter instead.
- The new WFS 2.0.0 querying also allows paged WFS requests which has impact on a number of hooks:
  - pydov.util.hooks.AbstractReadHook.wfs_search_init` now has a single argument params with all the parameters used to initiate the WFS search.
  - pydov.util.hooks.AbstractReadHook.wfs_search_result now has two arguments number_matched and number_returned. Since there can be multiple (paged) WFS results per search, this hook can now be called multiple times per search query. It can also be called simultaneously from different threads.
  - pydov.util.hooks.AbstractReadHook.wfs_search_result_received is affected in a similar manner: this too can now be called multiple times per search, simultaneously from different threads.
  - pydov.util.hooks.AbstractInjectHook.inject_wfs_getfeature_response is affected as well. This too can now be called multiple times per search, simultaneously from different threads.
New features
- Add support for WFS paging, allowing larger queries. It is now possible to execute queries larger than the WFS response feature limit of 10000 features without running into a FeatureOverflowError. (#194)
  
  Please be kind to our infrastructure and only request the data you need.
- Add support for repeatable sessions, allowing recording and replaying of pydov sessions. More information and use cases can be found in the documentation. (#224)
Fixes and improvements
- Fix parsing of datetime fields.
- Replace unparseable data with NaN and issue a warning. (#368)
- Raise MetadataNotFoundError when remote metadata fails to be downloaded or parsed.
- Populate custom fields in all circumstances. (#379)

Natural Resources - Soil and Land - Python
Published by Roel about 3 years ago

pydov - v2.2.3

Fixes and improvements
- Following the 0.28.1 OWSLib security release, disable XML entity resolution when using lxml's XMLParser.

Natural Resources - Soil and Land - Python
Published by Roel about 3 years ago

pydov - v2.2.2

Fixes and improvements
- When data received from DOV fails to be parsed by pydov, set it to NaN and issue a warning instead of crashing.

Natural Resources - Soil and Land - Python
Published by Roel over 3 years ago

pydov - v2.2.1

Fixes and improvements
- AquiferEnumType has been replaced with AquiferHCOVv1EnumType in both Grondwaterfilter and HydrogeologischeStratigrafie.

Natural Resources - Soil and Land - Python
Published by Roel over 3 years ago

pydov - v2.2.0

News
- This version adds support for Python 3.10.
- This version drops support for Python 3.6.
- This version is supported on Python 3.7, 3.8, 3.9 and 3.10.
New features
- Add new object types for soil data, including:
  - Soil depth intervals (bodemdiepteintervallen)
  - Soil classifications (bodemclassificaties)
- Added Fractiemeting as a subtype to the Bodemobservatie type.
- Renamed glauconiet to glauconiet_totaal in Grondmonster and added example on how to retrieve detailed glauconite values.
- Added support for proxy server autodiscovery using PAC.

Natural Resources - Soil and Land - Python
Published by Roel about 4 years ago

pydov - v2.1.0

News
- This version adds support for Python 3.9.
- This version is supported on Python 3.6, 3.7, 3.8 and 3.9.
- We are proud to be a part of the growing pyOpenSci community promoting open and reproducible research.
New features
- Add support for location-based searching using vectorfiles (f.ex. Shapefile, Geopackage) and Geopandas dataframes.
- Add new object type for groundwater permits (GrondwaterVergunning)
- Add new object types for soil data, including:
  - Soil sites (Bodemsite)
  - Soil plots (Bodemlocatie)
  - Soil samples (Bodemmonster)
  - Soil observations (Bodemobservatie)
Fixes and improvements
- Fix bugs that occur when the XML webservice is unavailable, i.e. prevent caching errors and return stale data if available.
- Retry failed network requests to make pydov more resilient to bad network connections.
- Switched from the main DOV WFS endpoint to workspace-level endpoints, this is more efficient and allows a cleaner codebase.
- Add the start_interpretatie_mtaw field to the interpretatie types.
- Add the mv_mtaw field to the Sondering type.

Natural Resources - Soil and Land - Python
Published by Roel about 5 years ago

pydov - v2.0.1

Fixes and improvements
- Fix the z field of the Sondering type, it is replaced by lengte and diepte following the DOV XSD schema update.

Natural Resources - Soil and Land - Python
Published by Roel about 5 years ago

pydov - v2.0.0

News
- This version drops support for Python 2.7 and Python 3.5.
- This version is supported on Python 3.6, 3.7 and 3.8.
Fixes and improvements
- Fix the korrelvolumemassa, volumemassa and watergehalte fields of Grondmonster type.
- Add the mv_mtaw field to the GrondwaterFilter type.
- Extend the hooks system and distinguish between read and inject hooks. The xml_requested hook has been removed in favor of xml_received.
- Generate stable WFS GetFeature requests, allowing f.ex. hooks to reuse cached responses.
Development-only updates
- Remove some code duplication between pydov and OWSLib.
- Simplify test fixtures setup.
- Remove duplicate docstrings to simplify the codebase.
Documentation-only updates
- Add introductory tutorial.
- Add a tutorial on how to use a WFS geometry as location query.
- Update development installation instructions.
- Update folium examples to support the latest pyproj version.
- Add extra Binder links on top of each tutorial.
- Improve charts by including a title and axis labels.
- Improve README by adding dataframe output.

Natural Resources - Soil and Land - Python
Published by Roel almost 6 years ago

pydov - v1.0.0

News
- This version is promoted to Stable.
- This version is the last to support Python 2.7.
Fixes and improvements
- Fix the PropertyInList and Join query operators.
- Increase the default request timeout to 5 minutes to enable larger WFS queries.
- Retype the meetnet_code field of GrondwaterFilter from integer to string.
- Pin the dependencies to keep explicit Python2 support.
Development-only updates
- Make the DOV base URL configurable to be able to test against the DOV testing environment.

Natural Resources - Soil and Land - Python
Published by Roel over 6 years ago

pydov - v0.3.0

News
- This version is promoted to Beta.
- This version adds support for Python3.7 (next to 2.7, 3.5 and 3.6)
New features
- Add new object type for Borehole samples (grondmonsters)
- Add new object type for Groundwater samples (grondwatermonsters)
- Add new object type for Informal hydrogeological stratigraphy (informele hydrogeologische stratigrafie)
- Add support for runtime object type customization (pluggable types) allowing full control of the output dataframes
- Add support for limit (max_features) when searching: this allows to explore the results of a query easily
- Add support for sorting when searching, allowing to retrieve f.ex. the deepest borehole etc.
Fixes and improvements
- Fix 'mv_mtaw' field of GrondwaterFilter, it is renamed to 'start_grondwaterlocatie_mtaw'
- Output dataframe columns are now in the order provided in return_fields, if available.
- The PropertyInList and Join query operators now work with single-item lists and dataframes too.
Documentation-only updates
- Fix DOI badge and Zenodo link: always link to the latest release

Natural Resources - Soil and Land - Python
Published by Roel over 6 years ago

pydov - v0.2.1

Fixes and improvements
- Fix download of Feature Catalogues from the new DOV Geonetwork 3.6 instance.

Natural Resources - Soil and Land - Python
Published by Roel over 6 years ago

pydov - v0.2.0

New features
- Add new object type for Quaternary stratigraphy (Quartair stratigrafie)
- Add support for using Join using a different column name: Join(df, on='...', using='...')
- Add 'filterstatus' and 'filtertoestand' to Peilmeting subtype of GrondwaterFilter
Fixes and improvements
- Fix search for GrondwaterFilters (update for WFS service changes regarding filternr)
- Fix 'Methode' field of Peilmeting subtype of GrondwaterFilter
- Exclude empty filters (i.e. Put without Filter) from GrondwaterFilterSearch
- Improve performance by using parallel processing and connection pooling
Documentation-only updates
- Update contributing guidelines

Natural Resources - Soil and Land - Python
Published by Roel almost 7 years ago

pydov -

Natural Resources - Soil and Land - Python
Published by Roel almost 7 years ago

pydov -

This minor patch adjusts only the deployment setup itself.

Natural Resources - Soil and Land - Python
Published by stijnvanhoey almost 7 years ago

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Open Sustainable Technology

Recent Releases of pydov

pydov - v4.0.0

pydov - v3.3.1

pydov - v3.3.0

pydov - v3.2.0

pydov - v3.1.0

pydov - v3.0.0

pydov - v2.2.3

pydov - v2.2.2

pydov - v2.2.1

pydov - v2.2.0

pydov - v2.1.0

pydov - v2.0.1

pydov - v2.0.0

pydov - v1.0.0

pydov - v0.3.0

pydov - v0.2.1

pydov - v0.2.0

pydov -

pydov -