Releases | Open Sustainable Technology

HydPy - HydPy 6.1.4

Again, we encountered cases where HydPy-Musk-MCT calculated nan values for low-flow situations. This time, these were caused by negative estimates of the reference discharge. We fixed this problem by simply resetting negative values to zero (see Calc_ReferenceDischarge_V1 for an example).

Hydrosphere - Freshwater and Hydrology - Python
Published by holgeralbert about 1 month ago

HydPy - HydPy 6.1.3

This bugfix release improves how main models share information with their sub-submodels. Previously, some data was not passed as deep copies (see 54a5b7dec72422ffafb78f7a83abb2e77a688cc8), which could result in wrong assumptions of sub-submodels on, for example, which hydrological response units of the main model consider interception processes.

The previous bug could impact simulation results. However, only those projects could be affected that use a main model that defines a specific "prepare method" (for example, prepare_interception), a submodel that does not specify an equivalent method, and a sub-submodel that uses the information provided by this method. Such combinations can currently only occur when HydPy-Evap is involved and should be rare. Hence, it is unclear if any real projects have been affected.

Hydrosphere - Freshwater and Hydrology - Python
Published by holgeralbert 2 months ago

HydPy - HydPy 6.1.2

We encountered cases where HydPy-Musk-MCT calculated nan values for extremely small reference discharge values. We could fix this problem by increasing the precision of the iterative search for the corresponding reference water level (based on the Pegasus method). Hence, this bugfix release should not only avoid the nan value problem but might also result in slightly different (typically slightly more precise) results during low flow periods compared to previous releases. Please have a look at #162 for more information.

Hydrosphere - Freshwater and Hydrology - Python
Published by holgeralbert 3 months ago

HydPy - HydPy 6.1.1

The function create_projectstructure introduced in HydPy 6.1.0 was only of limited use because one still had to manually create the project's subdirectories (for example, projectroot/network/default). With HydPy 6.1.1, the different FileManager subclasses do this automatically if their base directory (for example, projectroot/network) is still empty. See (the end) of the documentation on the currentdir property for an example.

Hydrosphere - Freshwater and Hydrology - Python
Published by holgeralbert 7 months ago

HydPy - HydPy 6.1.0

With HydPy 6.1, we switch to a tighter release cycle. Compared to HydPy 6.0, there are a few breaking changes, but those require, at most, little client code adjustments (ask you type checker) and do not change any functionality. Hence, the transition from 6.0 to 6.1 should be smooth.

Here is the list of the most notable new features, documentation improvements, and fixes.

New features:

The different file managers (for example, the ConditionManager) now print a note when they create or remove a directory. This feature can be turned off by setting printprogress to False.
The new "script function" xml_validate allows to check if an XML workflow file fits to XSD schema definition file of the relevant HydPy version from the command line.
Reading time series from or writing time series to NetCDF files via the related methods of the classes HydPy, Elements, and Nodes now works "automatically", meaning one has not to bother with manually opening and closing NetCDF reader and writer objects anymore.
The option reprdigits now controls the number format for writing ASCII time series files. By default, all numbers are written in scientific notation (consistent with HydPy 6.0). If you set reprdigits to two, for example, all numbers are written in the decimal form with at most two decimal places
Class ConditionManager now has the option prefix for changing the prefix of the automatically determined, time-dependent condition directory names. This feature eases, for example, to read the initial conditions of different ensemble members from separate directories
All subelements of <conditions_io> in XML workflow files are now optional. Making the XML elements <inputpath> and <outputpath> optional was necessary because otherwise, the new element <prefix> would be generally impactless. Making the XML element <zip> optional was primarily for consistency but also for convenience, as its default value false should apply to most use cases.
When using HydPy from the command line, you can now select two alternative error styles (single_line and splittable) to squeeze printed error messages in single lines, which is more easily processable for tools like Delft-FEWS. The old behaviour is now named multiline and the default of the new errorstyle option. For more information, please look at the documentation for the central hyd.py script.
Function summarise_ncfile now automatically determines and prints the time grid of the analysed NetCDF file if it has the time reference attribute defined.
The new functions create_projectstructure and check_projectstructure create and check the basic directory structure of a HydPy project. By default, checking also happens automatically when initialising class HydPy. This feature can be turned off by setting checkprojectstructure to False.

Documentation improvements:

We added the Simulation section to the User Guide. It explains semi-comprehensively performing a simulation run via Python scripts or XML workflow files. It should build a bridge between the Quick Start's introductory remarks and the Reference Manual's extremely detailed explanations.
The new Creating an Environment section of the Installation Instructions introduces to using virtual environments for the robust installation of, for example, different HydPy versions on the same computer.

Fixes:

Reading initial conditions files with initial condition data already in RAM (for example, resulting from a previous simulation run) could result in unnecessary trimmings of the read values. The user was warned but had to solve the problem himself (for example, by deleting the data first and rereading the files afterwards). We fixed this issue by applying the "postponed trimming" strategy already implemented for resetting data via dictionaries.

Hydrosphere - Freshwater and Hydrology - Python
Published by holgeralbert 7 months ago

HydPy - HydPy 6.0.1

In HydPy 6.0.0, when iterating through the individual Timegrid instances handled by an instance of class Timegrids, eval_ was skipped. We fixed this in commit a32142d4ff4e52af823963f3151155bbeec355ea.

Hydrosphere - Freshwater and Hydrology - Python
Published by holgeralbert 8 months ago

HydPy - HydPy 6.0.0

We are happy to announce the release of HydPy 6.0, which comes with many documentation improvements, lots of new models, and, most importantly, a boost in flexibility in combining different model components.

Documentation improvements

We gave the documentation a new structure. Beginners will find it convenient that it now differentiates between a User Guide, which offers a basic introduction to HydPy, and a Reference Manual, which explains everything in depth. We included introductory tutorials (and plan to add tutorials that cover more complex use cases soon) based on the reworked HydPy-H-Lahn example project.

Submodel concept

Previously, one could easily combine models of different "families" in one project (for example, of HydPy-H(BV) and HydPy-L(ARSIM). However, no out-of-the-box mechanism existed for creating new combinations of different model components. Therefore, HydPy 6.0 introduces the Submodel concept. It allows, for example, to set up an HBV-like model that calculates evaporation similar to LARSIM.

So far, the new Submodel concept has been widely rolled out for meteorological factors like global radiation and evapotranspiration, and it at least partly covers topics like runoff concentration and flow formulas. We will modularise more model components in future HydPy versions. See the Submodel Graph in the new Model overview section for a complete list of the currently possible (and advisable) model compositions.

New models

At first sight, HydPy 6.0 comes with a multitude of new models. However, many are just new submodels that we extracted from already existing main models (the new Definitions section explains these terms in detail). For example, HydPy-Evap-PET-HBV96 stems from HydPy-H-HBV96.

Nevertheless, there are still some truly new models worth mentioning:

HydPy-Evap-PET-AMBAV-1.0 calculates potential evapotranspiration in agreement with the AMBAV 1.0 model of the Deutscher Wetterdienst.
HydPy-Dam contains three new models for modelling pumps and sluices.
The new model family HydPy-GA is supposed to provide Green-Ampt infiltration models. The first available models implement the GARTO method, which can be used both as a main model to simulate infiltration excess overland flow for complex rainfall events independently and as a submodel to refine the volume-based runoff generation approach of HydPy-L(ARSIM).
The new model family HydPy-SW1D introduces "hydrodynamic flood routing" to HydPy. It aims to robustly solve the 1-dimensional shallow water equations in river and channel networks. So, HydPy can now deal much better with catchments in lowland regions where hydraulic structures like weirs and sluices or tidal effects play a major role.

Model name changes

HydPy's model names were too cryptic, so we switched to more descriptive ones. For example, hland_v1 or HydPy-H-Land Version 1 is from now on hland_96 or HydPy-H-HBV96 (adoption of SMHI-IHMS-HBV96). You can find a complete list of all renamings here.

Hydrosphere - Freshwater and Hydrology - Python
Published by tyralla 8 months ago

HydPy - HydPy 6.0a0

This is the first pre-release of HydPy 6.0. Almost all of its new features have been implemented. Please look at issue #122 for the things still to come, which are mainly documentation-related.

We now support NumPy 1 and 2 (see issue #147). We included both in our test suites, but the now consistent usage of 64-bit integers (and floats) on all systems and all NumPy versions might cause unexpected inconveniences. Please open an issue if you encounter any problems.

Hydrosphere - Freshwater and Hydrology - Python
Published by tyralla 10 months ago

HydPy - HydPy 5.0.3

This release includes minor adjustments to make HydPy 5.0 compatible with older versions of NumPy (tested against NumPy 1.22).

Hydrosphere - Freshwater and Hydrology - Python
Published by tyralla over 1 year ago

HydPy - HydPy 5.0.2

This release contains minor technical improvements compared to 5.0.1 and, more importantly, some adjustments to recent site-package changes (e.g. of NumPy). Its wheels are also available for Python 3.11, the latest Python version.

Hydrosphere - Freshwater and Hydrology - Python
Published by tyralla over 1 year ago

HydPy - HydPy 5.0.1

Fix passing an application model's module to the constructor of class Rule of module calibtools. 62d6d9bd2d517f5f8e30fe1819e820306b8aca00

Hydrosphere - Freshwater and Hydrology - Python
Published by tyralla over 2 years ago

HydPy - HydPy 5.0.0

We are pleased to announce the release of HydPy 5.0. It includes technical improvements, freshly implemented hydrological models and extended data-assimilation support. Also, it marks the first step of our efforts towards a higher degree of model modularity, with which we strive to increase the compatibility of the process abstractions of different model families. In the following, we highlight the most notable changes.

First of all, one "feature" that does not relate to HydPy 5.0 directly but to a policy established during its development: We try to shift at least the most relevant content-related discussions between the core members to the GitHub issue system. Sometimes we discuss online; other times, we only document discussion results. This policy helps us to keep track of different ideas and how things evolved, and - as important - it allows others to trace the development of HydPy better and to participate.

New models and model improvements:

The new model H-Land (HBV96-SC) is a slight modification of the long-available H-Land (HBV96) model. Like H-Land (HBV96), H-Land (HBV96-SC) works largely like the conceptional HBV96 model but represents runoff concentration with a linear storage cascade instead of a triangular-shaped Unit-Hydrograph. We implemented the storage cascade as an array of states, allowing us to address it in state-based data assimilation efforts.
H-Land (HBV96-SC/PREVAH) combines concepts from HBV96 and PREVAH, which is also a successor of the original HBV model. All processes "above the soil" (input data correction, interception, snowmelt) and "inside the soil" (evaporation, generation of effective precipitation), as well as the handling of water areas, are identical with H-Land (HBV96) (and so with HBV96). Most processes "below the soil" agree with PREVAH (runoff generation and concentration). We intended to implement H-Land (HBV96-SC/PREVAH) to improve the drought and small summer-event simulation of H-Land (HBV96) with process equations suitable for state-based data assimilation methods. We are pretty happy with its performance so that it will soon replace the current low-flow forecasting model for the river Rhine of the German Federal Institute of Hydrology. #67
H-Land (HBV96-SC/COSERO) combines concepts from HBV96 and COSERO, which is another successor of the original HBV model. We developed it for the same reason and similar to H-Land (HBV96-SC/PREVAH), except that most processes "below the soil" agree with COSERO. In our first very preliminary studies in a few Rhine sub-catchments, its results were a little less convincing than those of H-Land (HBV96-SC/PREVAH), but we still need to examine it in more detail. #68
All members of the HydPy-H-Land model family now include the new land cover type SEALED, allowing for simulating surface runoff from sealed surfaces. #71
We improved the snow module of all HydPy-H-Land with the alpine regions of the river Rhine in mind. The changes include (1) a seasonally varying day-degree factor for considering the effect of the annual cycle of global radiation on snow melt, (2) the further subdivision of hydrological response units into snow classes for modelling the effects of small-scale snow-depth variability, and (3) the redistribution of snow due to gravitational and wind forcing. #70
We introduced the HydPy-Musk model family, providing Muskingum-like methods, which are finite difference solutions to the routing problem. Its member Musk (classic) replaces HydPy-H-Stream Version 1. It implements the "classic" Muskingum approach (relying on the storage coefficient k and the weighting coefficient x). Alternatively, it supports defining the working equation coefficients via the parameters lag and damp like HydPy-H-Stream Version 1 did in the style of the IHMS implementation of HBV96. Ambitious users can also calculate the coefficients themselves. The second available member is Musk (MCT), which implements a form of the Muskingum-Cunge routing method developed by Todini. Currently, it is hard-coupled to the Manning-Strickler formula, applied on a trapezoidal channel profile. #85
Until HydPy 4.0, our HydPy-Dam models did neither consider precipitation nor evaporation explicitly. Instead, the surrounding "land models" had to calculate these properties (because of our early efforts to strive for consistency with LARSIM). Now, all HydPy-Dam members take measured or calculated time series of precipitation and potential evaporation into account (for consistency with the old behaviour, one must currently supply time series with zero values). #51 #69
We added the HydPy-Exch model family, which opens new paths for coupling different model instances in one project. Its first member Exch (weir) allows for a gradient-based, bidirectional coupling of the inputs/outputs of other model instances and is inspired by the IHMS implementation of HBV96. So far, we have used it in combination with Dam (controlled lake) to model, for example, the connectivity of the Swiss lakes Morat, Neuchatel, and Bienne via relatively short canals, through which water can flow in both directions depending on the actual water level differences of the respective lakes. #69
The new model family HydPy-Meteo targets more fine-grained modularisation regarding meteorological "preprocessing". Meteo (global radiation, FAO) and Meteo (global radiation, LARSIM) calculate global radiation based on (possibly measured) sunshine duration and rely on the equations documented for the FAO grass reference evaporation method and the LARSIM model, respectively. Meteo (sunshine duration, FAO) and Meteo (sunshine duration, LARSIM) handle the inverse case and calculate sunshine duration based on global radiation (possibly stemming from meteorological simulations). This refactorisation allows, for example, to calculate potential grass evaporation with Evap (FAO) using LARSIM-like global radiation estimates or to calculate actual Penman-Monteith evaporation with HydPy-L-Land (Penman-Monteith, Knauf) using FAO-like global radiation estimates. #81
We added more methods for checking that our model implementations satisfy the water balance equation, especially for our HBV- and LARSIM-like models. All checks are part of the automatic test system, and we fully report their results in the online documentation.
We standardised the units of many variables. So, for example, we now generally use $hPa$ for pressure terms and $W/m²$ for energy fluxes. This change might introduce some deviations from the original model descriptions in the literature but helps to avoid mistakes when setting up projects including different types of models. #81
The new variable type FactorSequence allows for a better grouping of a model's different time series-related variables. We use it, for example, to separate state-like meteorological factors like air temperature from more flux-like variables like precipitation.

General and technical aspects:

We dropped support for Python 3.6 (which reached its end of life) and started supporting Python 3.10 (the most recent Python release) instead.
The processes for building (based on setuptools, not distutils) and testing (based on nox) are now up-to-date. One benefit for users is that creating HydPy wheels for alternative platforms becomes easy. #74
HydPy 5.0 is compatible with the HydPy-OpenDA adapter 1.0 and includes numerous changes to satisfy its new functionalities. Please see the adapter's documentation and release notes for further information.
Using NetCDF files now comes with a cleaner interface (mainly by removing of a few options that proved more irritating than helpful). Also, it now works both for left and right timestamps (meaning, the time points specified in a NetCDF file can relate to the start or the end of the respective time interval). #59 #84
Reading and writing data "just in time", which helps save RAM in massive projects, now relies on the NetCDF format and is much more efficient than our old approach. Another benefit is that there is no difference between "normal" NetCDF files and those required for reading or resulting from writing "just in time", which makes switching to the "just in time" approach as a project grows nearly effortless. #18 #60
HydPy 5.0 comes with some speed-ups in project initialisation. The most notable speed-ups relate to building the network between different model instances and preparing instances of the model family HydPy-Dam that define seasonally varying control rules.
Users can now select different interpolation approaches when defining x-y-relationships (for example, between water level and discharge). Besides the old (flexible but complicated) ANN-based interpolation, HydPy now supports different spline interpolation methods (including simple piecewise linear interpolation). Users can either calculate the interpolation weights themselves or apply a suitable spline method to their raw x-y-data. #52

Hydrosphere - Freshwater and Hydrology - Python
Published by tyralla over 2 years ago

HydPy - HydPy 4.0.1 (supplemental data)

In HydPy 4.0.0, one needs to trigger the automatical generation of the sequence alias modules inputs.py and outputs.py by hand.
Since HydPy 4.0.1, they are already included in our binary distributions.

Hydrosphere - Freshwater and Hydrology - Python
Published by tyralla over 3 years ago

HydPy - HydPy 4.0.0

It's been a while since we released HydPy 3.1, and much has happened since then. The following listing points out only the highlights of the improvements HydPy 4.0 comes with.

Regarding the model collection:

Our "old" LARSIM-like application models lland_v1 (Turc-Wendling, Degree day) and lland_v2 (External PET, Degree day) now include a wider range of the original LARSIM options (e.g. capillary rise).
The new model lland_v3 (Penman-Monteith, Knauf) includes LARSIM options frequently applied in the context of runoff forecasting in Germany (e.g. snow energy balance calculations after Knauf, 2006).
lland_v4 (Penman-Monteith, Knauf, snow interception) extends lland_v3 by the process of snow interception.
lstream_v001 (Kinematic Wave, Manning-Strickler) now keeps the water-balance perfectly. We achieved this by implementing LARSIM's "kinematic wave approach" instead of its original (problematic) Williams-like approach.
lstream_v002 (Kinematic wave, External rating curve) works like lstream_v001 but expects the relationship between water-stage and discharge to be calculated externally (e.g. by a hydrodynamical model).
dam_v006 (controlled lake), dam_v007 (retention basin), and dam_v008 (reservoir) are close emulations of the LARSIM models for representing lakes (SEEG), retention basins (RUEC) and reservoirs (TALS).
evap_v001 (FAO 56 PM) is our first pure "evaporation model" and implements the commonly applied FAO guideline for calculating potential evapotranspiration.
The "Converter" models conv_v001 (Nearest Neighbour), conv_v002 (Inverse Distance Weighting), and conv_v003 (Inverse Distance Weighting with External Drift) work technically like all other models but are no hydrological models. You can apply them for interpolating input data (e.g. temperature) or the output data of arbitrary models (e.g. the potential evaporation calculated by evap_v001).
wland_v001 (semi-distributed WALRUS model) is a (slightly extended) implementation of the WALRUS model, designed to simulate surface water fluxes in lowland catchments influenced by near-surface groundwater.
wland_v002 (experimental) is, as the name suggests, work in practice. So far, it simulates faster responses of the groundwater table than the original model due to more explicit considerations of the capillary fringe. We currently discuss other modifications to improve the WALRUS performance for "moderate-flat" catchments.

Note that, due to their extension, lland_v1 and lland_v2 now require some additional parameters.

Be aware that we might change some aspects of lland_v3 later. We will probably unify some parameter names and possibly even modify some equations.

General features:

We now test HydPy from Python 3.6 to 3.9 (we think about dropping 3.6, but postpone this for the sake of keeping compatibility with the arcpy library of ArcGIS).
The command "pip install hydpy" now installs HydPy and all its requirements in one step.
Our AppVeyor workflow now creates an "HydPy-Installer" based on Python 3.9 for Windows. This is for users who want to experiment with HydPy without installing a complete Python environment or want to have a robust executable for just executing certain tasks (e.g. flood forecasting).
We improved many features of the online documentation. Many new automatically generated links help to browse through the different pages (for example, to directly jump from a parameter to the methods/equation requiring this parameter). Additionally, we start providing the documentation of several HydPy versions in parallel (for now, version 4.0 and our current development status, the "master branch"; others will follow).
HydPy now provides some server functionalities. The reason is to simplify the online coupling to non-Python software like OpenDA (for more information, see the separate documentation of our OpenDA-Adapter).
We added and extended lots of "user-comfort" functionalities concerning calibration, evaluation, plotting, and much more.

Scientifical and technical aspects:

We finally achieved a test coverage of 100 %. Our Travis-CI workflow does not accept any changes with incomplete coverage. And everything is tested "publically" in the HTML documentation (using doctests). This is a really significant milestone, as it makes introducing bugs much harder and allows the hydrological community to check all implemented equations easily "by hand". We are not aware of any other hydrological modelling system with such an eager and transparent test system.
We improved the tables and figures showing the results of the model integration tests. We hope they now better guide us into the depth of such huge models as lland_v3 and help to understand their details and possible implementation deficits. Our Travis-CI workflow automatically recreates all figures for each new release, so you can always be sure they are up-to-date.
The source code is now consistently formatted with black.
We started to add type hints to the source code following PEP 484. These improve the readability for code-analysis tools (and often also for humans) and improve IDE functions like auto-completion and bug-analysis. So far, these cover most of the features directly accessible to the user. We continue working on this.
We increased the performance of HydPy, mainly regarding initialisation. Two examples are the "lazy importing" of optional site packages that are seldom required (finished) and the caching of pre-compiled byte-code (still room for improvements).

Most of the listed (and unlisted) improvements were implemented on behalf of the German Federal Institute of Hydrology and the Forecasting Centre of the German federal state of Saxony (Landeshochwasserzentrum).

Hydrosphere - Freshwater and Hydrology - Python
Published by tyralla over 3 years ago

HydPy - Fix warning messages

HydPy informs the user when trying to read missing or incomplete observed or simulated time series data into Node objects. Due to some recent changes, this feature became unreliable. Version 3.1.1 fixes this problem and adds explanations and tests to the online documentation, to (hopefully) prevent this from happening again.