GEB - 1.0.0b30

Implement evaporation for retention basins using a constant area based on maximum storage and a depth of 3 meters.
Added optional external discharge skill-score comparisons for Google Streamflow, GloFAS, and PCR-GLOBWB/Utrecht.
Expanded discharge metrics with original/modified KGE, KGE components, NSE, Pearson r², RMSE, and RRMSE.
Added skill-score maps, boxplots, external KGE comparisons, upstream-area diagnostics, and dashboard summaries.
Improved the interactive discharge evaluation dashboard, and remove the option for static dashboards.
Add a --method-arg KEY=VALUE option to geb exec (e.g., geb exec estimate_return_periods --method-arg run_name=default) and fix spinup and run discharge not being concatenated for return period estimation.
Add hydrograph shape methods for floods. Instead of assuming a triangular shape, the shape of the hydrograph can now be derived from historical GEB discharge.
Add a CLI option to run yearly mode multiple times (e.g., geb run-yearly --multi --n-runs 5) and write each run to its own output folder.
Add new option (and implement) that you can only simulate floods in a subset of the basins in the larger region.
Implement general method for setting up an alternative universe.
Make it possible to report data from the alternative universe.
Fix bug for coastal regions where due to "holes" in deltadtm for lakes and reservoirs while fabdem was not used because it was only used above 30 meters. Solved by instead always using fabdem by default and overwriting with deltadtm where available.
Remove redundant iteration from kinematic routing.
Add option to run and spinup "--skip-done". When turned on, we first check if the model was not already done and if so, don't run.
Set default of writing figures for SFINCS to false in reasonable default config. For example, writing the return period figures is about 10 times as slow as the actual calculation. Of course, users can set this to true whenever needed in their own config!
Implement custom and improved algorithm for river burning. Currently only works for the non-subgrid mode of SFINCS.
Add a --method-arg KEY=VALUE option to geb exec (e.g., geb exec estimate_return_periods --method-arg run_name=default) and fix spinup and run discharge not being concatenated for return period estimation.
Add hydrograph shape methods for floods. Instead of assuming a triangular shape, the shape of the hydrograph can now be derived from historical GEB discharge.
Make a new option for subbasin selection for flood simulations (new default): 'auto'. This automatically selects only subbasins that have their bankful discharge exceeded and only simulates flood events for those subbasins.
For return period maps, we simulate subbasins one by one. Each simulation includes the downstream subbasin as well. This downstream subbasin also has inflow from other rivers. In this update the downstream subbasins also receive water from sidestreams (all-year mean).

Hydrosphere - Freshwater and Hydrology - Python
Published by jensdebruijn about 1 month ago

GEB - v1.0.0b29

Load return-period flood maps from the spinup output folder (output/{spinup_name}/flood_maps/{return_period}.zarr).
Re-organize evaluate hydrodynamics to have all functions outside the method body.
Make destination earth API more robust (twice).
Use localtime rather that UTC time for logging.
Use Event tuple rather than dictionary to track events through model.
Refactor evaluation of hydrodynamics. There is now a new "evaluate_flood" that evaluates the flood observations set in the build process. In the future, "evaluate_hydrodynamics" should be removed entirely, but we first need to add the ability to add custom flood maps to the setup_flood_observations build method.
Several fixes for hydromt-sfincs 2.0.
Update to pandas 3.0. This update has been prepared for a while and tested, but there may be potential issues popping up due to different copy behaviour in pandas 3.0. See here for more details.

Hydrosphere - Freshwater and Hydrology - Python
Published by jensdebruijn about 2 months ago

GEB - v1.0.0b27

Make model building fully deterministic (#821).
Implement retry mechanism for 429 web error GLOPOP-SG data adapter. Zenodo rate limited making too many range requests. If we get rate limited now, we now go to sleep for a bit and try again later.
Further reduce memory use of reporter by using numpy views for identical time series data.
In a recent change, discharges were explictly set to nan in reservoirs (good). This led to a case where rivers had 0 river width in the hydrodynamic model, which led to raised errors. Now, we fix this by looking further downstream of reservoirs to find a valid discharge for estimating river widths. This is also better, because perviously the default alpha for river width was used.
Add geb tool merge command to merge outputs from multiple GEB cluster sub-models into a single merged model directory that can be evaluated directly with geb evaluate.
Add plot_skill_score_maps() to plot skill scores per station on a satellite basemap for each metric.
Update plot_skill_score_boxplots() with violin + boxplot panels for KGE, NSE, R², RMSE and RRMSE, including support for overlaying external reference model (e.g. LISFLOOD) results.
Add R², RMSE and RRMSE to discharge evaluation metrics at hourly, daily and monthly frequencies.
Stations with fewer than 5 years of paired observations are skipped in evaluate_discharge.
Lakes and reservoirs are now shown as dot markers in the interactive discharge evaluation map.
Include height above nearest drainage (HAND) for later use.

Hydrosphere - Freshwater and Hydrology - Python
Published by jensdebruijn 2 months ago

GEB - v1.0.0b26

Write new model version after each version update in geb update-version. This way, when the model is updated multiple versions ahead, and one of the updates fails, the version updates that succeeded are still "saved".
Recently, there have been a lot of supply chain effects, where packages contained malicious code or instructions. To avoid most of these issues, but at the same time get relatively new updates (including those with important security updates) we limit any package updates to packages that are at least 3 days old in the uv.lock file. Here we use uv's exclude-newer option.
Update routing to include retention basins. By default no retention basins are set (all -1), however a dataset can be passed to set up retention basins. These basins can retain water during flood peaks, and slowly release water during low-flow periods. Set up using setup_retention_basins.
Fix case where river discharge was 0 in waterbodies (OK), but led to division by zero error in determining alpha for river widths (#819).
Use MIRCA-OS crop calendars rather than MIRCA2000 (#813).
Pre-allocate numpy arrays in reporter for both time and data. This uses significantly less memory as compared to Python lists, and because arrays are pre-allocated they do not cause RAM issues very late in the run but immediately instead.
Include an array tracking household expected annual damages (based on adaptation status).
Support exporting household attributes (such as ead) using the reporter in run_yearly.
When multiple outflow basins were selected that are not coastal basins, it could happen that some basins were erroneously excluded. This is now fixed.

Hydrosphere - Freshwater and Hydrology - Python
Published by jensdebruijn 3 months ago

GEB - v1.0.0b25

Fix cases where subgrid elevation could be nan in coastal areas and DEM was not available. This ultimately leads to an error in the land surface model (propagating nans).
Include a delta approach to account for changes in precipitation and temperature under climate change in creating return period maps. To adjust forcing data to future climate, add the representative_forcing_year argument to setup_forcing in the build.yml to indicate the year for which you want to fast-forward the forcing data.
Fix a bug where update-version would not write newly created files to files.yml.

Hydrosphere - Freshwater and Hydrology - Python
Published by jensdebruijn 3 months ago

GEB - v1.0.0b24

Add mode: "off" option to setup_waterbodies to completely disable waterbodies, or mode: "lakes_only" or mode: "reservoirs_only".
Add --debug-method "method_name" to geb build to filter the build to only run the setup region and any other methods that are required to run that specific method. For debugging purposes only.
Update DeltaDTM adapter to download continent ZIP files, unpack them, and then save the unpacked files on disk. This avoids issues with temporary files.
Load GLOPOP-SG files directly in memory, rather than first writing to disk and then loading to memory.
Improve alignment of data that goes into land surface model, reducing memory contention. And various other optimizations in land surface model.
Fix "Inflation not set when model runs to final year of data that was created in build process" (#808).
Flood events could not and still cannot be simulated during the spinup. However, this was not clear and a funny error about missing files was raised. Now this situation is detected and a clear error is raised (#806).
Simplify and speed up reading of forcing data. Before we used chunks of only 24 hours and used complex async reading to make it fast. Now that we use much larger chunks and efficient masking, the asynchronous reading made things overly complex and error prone. Simplifying the reading (e.g., dropping the async reader) while optimizing the indexing makes the model 10-20% faster in the test region. The difference is negligible in large regions.
Switch to Ross methodology for vertical soil water balance. This improves stability model speed.
Switch to Côté-Konrad kappa-based method for calculating soil/water heat conductivity. This uses no power functions and is much faster to compute.
Use titles for names in GRDC discharge plots.
Several updates for visualization of discharge evaluation. Switch to different background, use consistent coloring, replace scatter plot with return period plot.
Large number of optimizations in LSM making the model significantly faster again.
Speed up sampling from subgrid to HRUs using multi-processing.
Support 0 delay of runoff concentration.
Set variable runoff concentration variable to 1.0 by default (needs spatial discretization).
Improved layout of profiling txt file.
Export less data by default.
Combine various reporter outputs more efficiently.
Use Zarr ScaleOffset and CastValue codecs instead of numcodecs (which are off-spec and will be deprecated at some point).
Return median metrics for KGE, NSE etc rather than mean.
Add FLUXNET (https://fluxnet.org) and GROW (https://zenodo.org/records/15149480) observations to build process. Not yet used in evaluation.
Export less data by default. If you miss any files that you relied on they can be explicitly turned on in your model.yml. See the reasonable_default_config.yml for references. This makes the model ~10% faster.
Combine various reporter outputs more efficiently reducing the amount of (duplicate) files exported. Naming conventions have changed slightly, but code in the evaluation functions was updated accordingly.
Use Zarr ScaleOffset and CastValue codecs instead of numcodecs (which are off-spec and will be deprecated at some point). Also they are faster.
Do not convert from grid to HRU and then back to grid for livestock water consumption.
Abstraction from industry is now assumed to be abstracted from larger rivers only. If we let industry abstract from each grid cell that has any industry, the industrial users abstract water from very small rivers, which also leads to very high groundwater abstraction in those cells because the demand is not satisfiable from the river. This is highly unrealistic. Therefore, we define abstraction areas based on the river network. Each abstraction area is associated with a river of shreve stream order above a set threshold. All water demands from industry are essentially transferred downstream to the river of the abstraction area, and abstraction is assumed to occur from that river.
All other configuration options that essentially tried to do some of these things above per study area (like custom abstraction) are removed now. Hopefully we can simply reduce the need for configuration options with better defaults!
Change units that industry and livestock water demand return (now m3/day, was m/day).
Add surface area ratio to setup_geomorphology.
Use surface area ratio to set variable runoff.
Improvement of variable names and unit conventions in land surface model.
Limit infiltration to most restricted layer and set a simple surface crust to limit infiltration to 20% of saturated conductivity for top layer. Can be improved upon.
Export evapotranspiration for locations with flux towers.
Improve visualisation of discharge_evaluation.html. Several things to make it look better, but also make it roughly 33-40% of original size while making it faster loading.
Report monthly discharge evaluation values too.

Hydrosphere - Freshwater and Hydrology - Python
Published by jensdebruijn 3 months ago

GEB - v1.0.0b23

Add documentation, repository, and issue tracker links to pyproject.toml (#797).
Update license specification in pyproject.toml to follow PEP 639.
Re-support running with yearly timestep (fix several small bugs with variables not being available).
Add create_plots to setup_discharge_observations. This is because the plots are quite large and take long to generate. Default is false.

Hydrosphere - Freshwater and Hydrology - Python
Published by jensdebruijn 4 months ago

GEB - v1.0.0b22

Remove option to auto-fix build order (now build order should be much more consistent with reasonable_default_build.yml) and fix associated tests.

Hydrosphere - Freshwater and Hydrology - Python
Published by jensdebruijn 4 months ago

GEB - v1.0.0b21

Remove ncpus config option from SFINCS. CPU count is now always determined automatically from the SLURM environment or system.
Simplify assigning of crops and irrigation type in build process. Fix bug where sometimes irrigation type was not found.
Remove setup_irrigation_sources from build process as it is not needed anymore.
Fix basin delineation for endorheic basins.
Fix division by zero error for reservoirs that have no long term inflow.
Reduce RAM usage of reading GTSM data.
Table-like data is exported as parquet rather than csv (saves lots of space on disk).
Better compression of reported data. Most importantly, floats are now bitrounded with a maximum error of around 0.01%. In one test, the hourly discharge is now 168MB instead of 487MB.
In reported spatial data, the time dimension is now the last dimension. This allows for better compressibility (because of spatial auto-correlation). If you make plots and read the data with xarray you likely won't notice. If you read the data in funky ways with numpy, you may need to adapt some scripts.
Report discharge for outflow points of all rivers instead of just the outflow at the end of the basin with _outflow_points is set to true in the report.
Use the reported time series tables instead of grid when setting up the SFINCS models and estimating return period values.
Do not report discharge grid data by default in example (saving lots of disk space). You can re-enable this manually when you need it.
Remove setting up SFINCS model from gridded data directly. Not needed anymore (see above).
Fix all typing issues.
Enable better compression for any table-like data. Mostly targets GTSM data that is now better compressed.
All agricultural insurances premiums and insured yields are now determined by a new insurer agent named "insurers". The adaptation itself is still within crop_farmers. For index and precipitation insurance functions have been added to estimate candidate spaces for the potential contracts have been added. These estimate strike, exit and rates from the index and income/loss data. Additional insurers functionalities can be added to this agent now.
A hydrological year start month parameter has been added. This is a simple variable under "general" in the model.yml that indicates which month of the year the hydrological year starts. The base variable is 1 (january). Crop_farmers, livestock_farmers, market reservoir_operators, waterbodies and insurers take yearly actions at the start of the hydrological year.
setup_waterbodies has a new functionality where it can create reservoir command areas just based on the hydrology and reservoirs using the calculate_command_areas setting.
Bug fix where unify_crop_variants did not work for farmers for crop rotations with more than 1 crop.
self.var.adaptations is now a boolean array. Run geb update -b build.yml::setup_farmer_crop_calendar for it to become boolean.
setup_SPEI and setup_pr_GEV had issues with the grids not properly being assigned the actual GEV values. Rerun geb update -b build.yml::setup_SPEI and geb update -b build.yml::setup_pr_GEV to update
Fix division by zero error for reservoirs that have no long term inflow.
Fix basin delineation for endorheic basins.
Table-like data is exported as parquet rather than csv (saves lots of space on disk).
Better compression of reported data. Most importantly, floats are now bitrounded with a maximum error of around 0.01%. In one test, the hourly discharge is now 168MB instead of 487MB.
In reported spatial data, the time dimension is now the last dimension. This allows for better compressibility (because of spatial auto-correlation). If you make plots and read the data with xarray you likely won't notice. If you read the data in funky ways with numpy, you may need to adapt some scripts.
Report discharge for outflow points of all rivers instead of just the outflow at the end of the basin with _outflow_points is set to true in the report.
Use the reported time series tables instead of grid when setting up the SFINCS models and estimating return period values.
Do not report discharge grid data by default in example (saving lots of disk space). You can re-enable this manually when you need it.
Remove setting up SFINCS model from gridded data directly. Not needed anymore (see above).
Fix all typing issues.
Reduce RAM usage of reading GTSM data.
Enable better compression for any table-like data.
Save GTSM data as zarr with fixedscaleoffset and delta compression. Also adapt GTSM readers in model accordingly.
Remove all local caching during build to save disk space.
Enable automatic delta compression of time coordinates in zarr files.
Reduce RAM usage for models with a complex coastline.
Fix for differently sized crop maps from MIRCA-OS that led to issues where maps in regions where some maps did not overlap.
Several optimizations that make the model faster: faster grid-conversions, faster forcing interpolation.
Some fixes so that the evaluation functions read parquet files rather than the old csv files.
Fix for interpolation of MIRCA-OS crop data (#765)
Make the example build.yml inherit from a new 'reasonable_default_build.yml', allowing seamless updating unless custom settings are used.
Update pyflwdir to 0.5.11, which has caching of numba functions. Also thus allows removing of custom cached functions in routing.py.
Fix bug where river widths could be negative in rare cases. This clearly raised an error in the model run, so it doesn't affect any model that ran normally (#770)
Fix bugs in DeltaDTM: 1) tiles were not found as a buffer was not present around the coastal mask 2) for large coastal regions, the tiles were deleted, solving [#783]
Different updates to Global Exposure model (GEM), most importantly a detailed mapping of name changes between GEM and GADM
Fix bug where insurance_active tuple was always truthy; insurance check now uses any(insurance_active) in crop farmers (https://github.com/GEB-model/GEB/issues/790).
Fix wrong config key in livestock_farmers.py: was reading from agent_settings.town_managers instead of agent_settings.livestock_farmers. Since no config was actually used, this didn't have an effect on the model run.
Fix accounting bug in get_current_storage() where topwater was counted twice.
Replace efficiency [0–1] with explicit return_flow (m/day) in industry and livestock agents. The to_grid conversion for industry and livestock water demand is now performed inside update_water_demand(), so both agents return grid-scale arrays directly.
Speed up to_grid by parallelizing and simplifying because only weightedmean was used.
Split get_current_storage() into five sub-methods: get_landsurface_storage_m3, get_overland_flow_buffer_storage_m3, get_routing_storage_m3, get_waterbodies_storage_m3, get_groundwater_storage_m3 for better profiling and clarity. Also sped up some of the functions using numba.
Cache current_time in set_timestep() to avoid recomputing date every call.
Make saving of store and finalization of reporting multithreaded.
Refactor Reporter.process_value() into helper methods (_write_grid_hru_to_zarr, _apply_grid_hru_function, _write_agents_to_zarr, _apply_agent_function) for better readability and profiling.
Adapt plot_discharge in evaluate to work with timeseries instead of grid.

To support this version:

First of all it is HIGHLY RECOMMENDED to remove your own build.yml and replace it with the one in the examples. See geb/examples/geul/build.yml. This build.yml inherits from a new reasonable_default_build.yml (see geb/reasonable_default_build.yml). This will drastrically reduce the number of manual updates you need to do in the future. If you made any changes to your build.yml relative to the example, you can only keep those methods in your build.yml, which will then override the default ones in the reasonable_default_build.yml. If you want to keep the current setup, you need to remove setup_irrigation_sources from build.yml.

The following should be run automatically with geb update-version, but if you want to do this manually:

Re-run setup_SPEI: geb update -b build.yml::setup_SPEI.
Re-run setup_pr_GEV: geb update -b build.yml::setup_pr_GEV.
Re-run setup_farmer_crop_calendar: geb update -b build.yml::setup_farmer_crop_calendar.
Re-run setup_gtsm_station_data: geb update -b build.yml::setup_gtsm_station_data.

Hydrosphere - Freshwater and Hydrology - Python
Published by jensdebruijn 4 months ago

GEB - v1.0.0b20

Completely removed the region_subgrid. This subgrid was very large and led to several issues, including using lots of memory during the build. By refactoring the farms setup, this could be removed completely. This doesn't affect the model run as it never used it. Only internally in the build.
Refactored setup farms from lowder and created a test. Non-lowder datasets are not supported anymore. This will be added back later in the simplified setup when it is required for a specific purpose.
Write a custom zarr writer that is able to write chunked data and adapt all build methods to work with this.
Made numerous changes throughout the build to reduce memory usage. No content changes.
Set fill depressions to False by default in build. This option uses too much memory for large areas. If needed this can be build in again at the hydrodynamics stage.
Remove old data catalog entirely, and all references to it. Rename NewDataCatalog to DataCatalog.
Optimize GTSM data catalog, now pre-processing to zarr files.
Improve CLI help so geb evaluate --help list the available evaluation methods.
Fix issue in the enthalpy calculations. Previously there would be 0 infiltration even when only part of the soil was frozen. In addition, rainfall that didn't infiltrate never warmed the soil (if soil is colder..) which led to situations with too much frozen soil, too much runoff and too much discharge in winters.
Made quite a few plots and exporters for the water balance plotting. Note that not all plots show a correct balance yet. This is highly likely not due to actual balance errors (they are checked in the running model) but because we don't yet plot the right variables. To be continued..
Remove support for include_spinup in the evalution. This option was supported sometimes and sometimes not, which led to silent ignores and general over complications. It is still possible to run the evaluate for the spinup (only) by using the run_name: geb evaluate --run_name spinup.
fix reforestation water balance flux, option 1 route excess to topwater, option 2 source from topwater. This depends on how soil behaves at first time step when forests are planted.
For large scale (multiple basins) only: build stats are now written to CSV files under build_memory_stats/<cluster>.csv. Individual files are made for each basin cluster.
New command "geb clean" to reset and delete the data for a model, except the .yml files. Also works for multiple basin clusters/models.
The buffer size check fuction (check_buffer_size) is removed as this caused errors but is redundant.
Fixes in water circle displaying.
Update format for custom river discharge time series. See geb/examples/geul/data/discharge_observations
Make full integration test, now checking discharge with observed discharge in the test_run.
Fix recent regression where water demand for households was set to 0 except on January 1st.
Include evaluation tests in the test_run for simplicity.
Include global Huizinga curves as alternative to local Endendijk in build and reasonable default config.
Move evaluation of hydrodynamics to seperate file.
Also calculate discharge metrics at daily frequency if only hourly data is available.
Added auto-update for build methods.
Improve memory usage of setup_soil (hopefully)
Update to Python 3.14.4.

To support this version:

Run setup_global_damage_model: geb update -b build.yml::setup_global_damage_model.
Run setup_local_damage_model: geb update -b build.yml::setup_local_damage_model.
Included setup_subnational_income_distribution to also setup initial subnational income distribution parameters for simulating migration decisions.
Update to Python 3.14.4. Ensure that you update your uv first (uv self update).

Hydrosphere - Freshwater and Hydrology - Python
Published by jensdebruijn 4 months ago

GEB - v1.0.0b19

Add option for filling and raise out of bounds error for sample_from_map.
Activate dynamic river widths during spinup. During the first years of spinup there may be some small balance errors, but they will resolve over time and in the run (when river width alpha and beta are stable).
Only re-calculate household water demand every year (performance).
Set SPEI calibration period to 1960-1990.
Reduce memory usage during build with custom clip that works with dask.

To support this version:

Add a new file called 'build_complete.txt' in your input folder. In future versions this file will be made automatically.
Re-run setup_hydrography: geb update -b build.yml::setup_hydrography.

Hydrosphere - Freshwater and Hydrology - Python
Published by jensdebruijn 5 months ago

GEB - v1.0.0b18

Add loggers to groundwater model and SFINCS models.
Close all open figures in SFINCS to reduce memory usage.
Several fixes in sfincs.py to avoid futurewarnings for pandas 3.0.
Compress forcing data to 1D. This makes the input folder significantly smaller (~50% depending on the area).
Reduce area that elevation and land use maps are written for reducing size on disk.
Remove self.buildings_centroid as attribute (appears not to be used).
Load in buildings as pandas df, only load geometry data for flood damage calculations.
Make filling of discharge gaps a lot more efficient (quite some reduction in run speed).
Make it possible to specify the number of cores using --cores. Default is all cores (no change).
Make an option to auto-fix the build order if it is incorrect.

To support this version:

Re-run setup_forcing: geb update -b build.yml::setup_forcing.
Re-run setup_SPEI: geb update -b build.yml::setup_SPEI.
Re-run setup_pr_GEV: geb update -b build.yml::setup_pr_GEV.
Re-run setup_buildings: geb update -b build.yml::setup_buildings.

Hydrosphere - Freshwater and Hydrology - Python
Published by jensdebruijn 5 months ago

GEB - v1.0.0b17

Synchronize start and end dates in reasonable default config and example.
Add .zenodo.json

Hydrosphere - Freshwater and Hydrology - Python
Published by jensdebruijn 5 months ago

GEB - v1.0.0b16

Cleanup logging situation in model. Now each method (except init-multiple) should created their own log file in the logs directory and no additional logs should be created.
Fix several warnings throughout model. And do not ignore some warnings globally.
Add custom and improved logging in calibration snakemake workflow.
In calibration, only run init and build if model.yml and build was not completed respectively.

Hydrosphere - Freshwater and Hydrology - Python
Published by jensdebruijn 5 months ago

GEB - v1.0.0b15

Switch back to Python 3.13 due to netcdf reading errors.
Switch liquid water in snow and snow water equivalent to float64 to avoid floating point imprecision in thick snow layers.
In case of water balance or enthalpy error export data for single cell that can be used to fix and test water balance seperately.

Hydrosphere - Freshwater and Hydrology - Python
Published by jensdebruijn 5 months ago

v1.0.0b14

Only create plots during forcing setup if specifically requested with new create_plots argument.
Combine code in forcing.py so that it is more easy to maintain.
Remove unused setup_land_use_parameters.
Pre-process GRDC data to zarr with chunks for faster future reads.
Add object size profile when speed-profile is used.
Add version of when build was made.
Switch to Python 3.14
Make land surface build process more efficient and cleanup. As part of this update, only the original land cover within the SFINCS regions is saved. Therefore, this now depends on setup_coastal_sfincs_model_regions.
Yield is now computed from actual evapotranspiration and potential evapotranspiration rather than actual transpiration and potential transpiration. This is in line with GAEZ documentation, and also fixes a divide by 0 error.
Refactor runoff concentration, and solve very small WB bug due to order of operations.
There is now a new check to check the data version against the model version. If there is a mismatch, an error is given and the user is suggested how to update to the new model version. This only works for fresh builds. If you want to force this behaviour on already existing builds, run geb update-version.

Hydrosphere - Freshwater and Hydrology - Python
Published by jensdebruijn 5 months ago

GEB - v1.0.0b13

combine fabdem loading of elevation and forcing for saving some data on disk
add memory profiler memray. Use option e.g., geb spinup --profile-memory
renamed speed profiler to --profile-speed
remove return statement from setup_forcing that was left behind from a debugging session

Hydrosphere - Freshwater and Hydrology - Python
Published by jensdebruijn 5 months ago

v1.0.0b12

Reforestation: add government forest planting policy and soil modification workflow.
Convert suitable cropland/grassland to forest; update soils and remove farmers.
Reorganized geb/hydrology/ by moving land surface-related modules (landsurface.py, evapotranspiration.py, interception.py, snow_glaciers.py, potential_evapotranspiration.py) into a new geb/hydrology/landsurface/ package.
Split soil.py into geb/hydrology/landsurface/water.py (soil hydraulic processes) and geb/hydrology/landsurface/energy.py (soil thermal processes).
Add Leaf Area Index (LAI) integration in soil net radiation calculation to account for canopy shielding and emission.
Refactor discharge observations to support dual-frequency (hourly and daily) data tables.
Rename generic Q_obs to discharge_observations across the codebase for clarity.
Add frequency labels (hourly/daily) to extreme value analysis and validation plot titles.
Allow model to run from 1960 onwards (raise clear error if earlier than 1960 is requested).
Update parse_demand in agents.py to backward and forward fill water demand data if it doesn't cover the entire model time range.
Update discharge observation processing to support hourly data and separate observations into hourly and daily tables.
Update hydrology evaluation to support both hourly and daily observation datasets.
Add build method to set up reforestation potential (and data catalog entry)
Add units to all data from setup_hydrography
Compute hillslope length based on drainage density
Fix: Also set nodata type in _FillValue when using reporter. This is now also correctly loaded with zarr.
Add geb tool rechunk to allow rechunking of dataset to space-optimized, time-optimized or balanced. Currently using some reasonable defaults, but if needed we can expand this with custom values.
Add CWatM water demand to new data catalog (and remove from the old one).
Add --profiling option to geb build/update/alter.
Fix: fix for farm sizes that are all on the high end of the distribution.
Fix: fix for regions with very large coastal areas beyond the riverine grid
Use figures path for sfincs model to save all figures
Switch to hourly values for extreme value statistics
Use maximum of one flood peak per week
In evaluate make a dataframe without missing timesteps and ensure that return periods are esimated on the same data for observed and simulated for comparison.
Fix: fix for regions with very large coastal areas beyond the riverine grid.
Fix: waterbody outflow is larger than waterbody storage (due to floating point imprecision).
Fix: Added Liechtenstein to trade regions list which allows the model to be built in the Rhine basin
Move MIRCA-OS to new data catalog.
Move aquastat to new data catalog.
Add OECD Income Distribution Database (IDD) to the new data catalog.
Move Coast-RP to new data catalog.
Add heat conductivity to deeper soil layers (still missing influence of water).
Consider soil heat flux in pennmann-monteith.
Turn of sensible and turbulent heat fluxes in case there is snow.
Include evaporative cooling and advective heat transport from rainfall.
Add a daily soil enthalpy balance check.
Generalize river snapping.
Setup example preprocessing for retention basins.
Fix: Add iso codes for GDL regions where those are missing
Fix: Fix error in GLOPOP due to regions with 17 columns, instead of 16
Fix: Fix missing age (65) in age distribution for households
use GDL regions (instead of GADM) for the income distribution parameters
Fix: fix bug of farms that are smaller than the subgrid size
Fix: fix bug of countries that are not in trade regions (GLOBIOM)
Raise error when progress.txt contains duplicates
Speedup pr_gev calculation in build.
Simplify report function arguments.
Report water balance evaluation plot to evaluate folder.
Save climate data in weekly chunks, also read in weekly chunks -> significant speedup (~15% is some tests).
Use full penman-monteith for setup_SPEI.

To support this version:

Re-run setup_hydrography: geb update -b build.yml::setup_hydrography
Re-name setup_mannings to setup_geomorphology and run setup_geomorphology: geb update -b build.yml::setup_geomorphology
Re-run setup_discharge_observations: geb update -b build.yml::setup_discharge_observations
Only in case of build errors (or later in spinup/run):
- re-run setup_household_characteristics: geb update -b build yml::setup_household_characteristics
- re-run setup_crops: geb update -b build.yml::setup_crops
- re-run setup_income_distribution_parameters: geb update -b build.yml::setup_income_distribution_parameters
- re-run setup_create_farms: geb update -b build.yml::setup_create_farms

Recommended:

Re-run setup_forcing and setup_SPEI for a significant speedup and better SPEI estimation: geb update -b build.yml::setup_forcing and geb update -b build.yml::setup_SPEI

Hydrosphere - Freshwater and Hydrology - Python
Published by jensdebruijn 5 months ago

GEB - v1.0.0b11

setup_soil_parameters is removed in favour of setup_soil for consistency.
Add download and processing for soil thickness data.
DeltaDTM is now also setup for the model region in setup_elevation.
Align SFINCS mask padding to the coarse grid so left and bottom edges snap to grid-size multiples.
Improve inflow, outflow, flood plains and some other things to improve flood risk maps.
Remove DeltaDTM and GEBCO for non-coastal regions.
Re-indexing of OBM buildings and creating one household agent per building (per default).
Support multiple inflow locations.
(Deep) copy model config on initializing model avoiding reference issues.
Filter clusters in geb init-multiple based on intersection with coastline if parsed as argument.
Updated the GLOPOP version (from GLOPOP_SG_V2 to GLOPOP_SG_V3) to resolve missing data in some GDL regions
Add option for variable runoff in infiltration
Simplify coastal model setup. No longer create multiple shapes of connected low elevation coastal zones.
Moves to new data catalog
FAOSTAT
GLOPOP-SG
UNDP Human Development Index
OSM open_street_map_land_polygons
Support custom DEMs
Read custom reservoirs and waterbodies from files instead of old data catalog.
Add LISFLOOD vegetation properties adapter with crop group number and leaf area index support in setup_vegetation.
Add required = True/False to all build_methods allowing checking of build methods at build start rather than erroring when finally running the model.
Combine setup_crops and setup_crops_from_source.
Use LAI to set interception and compute crop factors for forest and grassland.
Use GTSM station data to get sea level rise for creating (future) coastal flood maps.
Add MIRCA2000 unit grid and crop calendar entries to the new data catalog and use them in crop calendar setup.
Move superwell data to new data catalog.
Switch MERIT Hydro dir/elv datasets to the global cache with a local fallback copy for offline access.
Change MERIT Hydro to use local GeoTIFF tiles directly instead of intermediate Zarr files.
Make trade regions inspired by globiom regions and load from file rather than data catalog.
Move osm land polygons to new data catalog
Add Global Exposure Model and GADM v2.8 to the datacatalog to assign building damages
Assign damages categories of the Global Exposure Model to the building geodataframe.
Calculate building damages both for structure and content using separate vulnerability curves for structure and content.
Check which MeritHydro files are present on the shared IVM datadrive. Ignore tiles that are not present in build as these are in the ocean.
Adjust wind speed computation to use FAO56 specifications.
Added a gadm_converter dictionary mapping incorrect GADM names to corrected versions in the global exposure model data adapter.
Moved global exposure model to global cache to deal with request limits (only 60 per hour when unauthenticated, just to prevent this becoming an issue)
Moved setup_buildings to its own function for quicker updating building attributes after changes.

To support this version:

Rename setup_soil_parameters to setup_soil in build.yml
Re-run setup_soil: geb update -b build.yml::setup_soil and setup_household_characteristics: geb update -b build.yml::setup_household_characteristics
Re-run setup_coastal_sfincs_model_regions: geb update -b build.yml::setup_coastal_sfincs_model_regions
Remove setup_low_elevation_coastal_zone_mask from you build.yml
Add setup_buildings to your build.yml
Models for inland regions need to be rebuild if floods need to be run
Re-run setup_gtsm_station_data: geb update -b build.yml::setup_gtsm_station_data to regenerate gtsm/sea_level_rise_rcp8p5 using the new GTSM station data.
Re-run setup_gtsm_water_levels: geb update -b build.yml::setup_gtsm_water_levels
Re-run setup_buildings: geb update -b build.yml::setup_buildings
Setup cdsapi for gtsm download, see instruction here: https://cds.climate.copernicus.eu/how-to-api
Rename setup_crops_from_source to setup_crops and use source_type rather than type (which is a reserved keyword in Python).
Add and run setup_vegetation to build.yml. A good place is for example after setup_soil.

Hydrosphere - Freshwater and Hydrology - Python
Published by jensdebruijn 6 months ago

GEB - v1.0.0b10

Coastal inundation maps are now masked with OSM land polygons before writing to disk.
Add documentation for modules, variables and routing.
Return period maps are now calculated per subbasin rather than using the whole map and making complicated calculation groups.
Flood maps of varying spatial domains can now be merged into one return period map.
Add llms.txt, llms-full.txt
Add MCP server for interacting with natural language (very much in beta)
Add a prompt to installation docs for setting up geb using llm agents
Update and simplify installation docs
Write documentation for spinning up and running models
Fix rare out of bounds values in ERA5 data that led to undefined behaviour due to compression and decompression roundtrip
Require extra_dims_names to be set in DynamicArray and update model in places where it was not set
Fill holes in subbasin maps by deriving subbasin maps directly from rivers ourselves. This also makes the original subbasins dataset not needed anymore.
Extend rivers to end up exactly in the ocean rather than the cell just before
Enable return period maps for subbasins that discharge into the ocean, including several bugfixes for this.
Allow exporting of hourly values from reporter
Add initial soil temperature. Now still simplified but better than having no soil temperature.
Includes soil suction into the model using an approximation of the Green-Ampt equation.
Use Green-Ampt rather than VIC for infiltration.
Implement interflow
Limit drainage to groundwater to conductivity of groundwater top layer
Fix for flood risk maps which could not be run if river was not included in grid but had upstream areas
Fix that downstream outflow area was not included with new subbasins
Renamed new_data_catalog to data_catalog and data_catalog to old_data_catalog

To support this version:

The model must be rebuild from scratch

Hydrosphere - Freshwater and Hydrology - Python
Published by jensdebruijn 7 months ago

GEB - v1.0.0b9

Updated numba to 0.63. This version fixes an error where changes in sub-functions were not always correctly detected when using caching behaviour.
Add a new option for flood models. We now auto-detect whether a change in the code or model input is made. If no change in the model or model input, we do not rebuild the SFINCS model. This removes the option force_overwrite for sfincs models.
Fix JSON serialization error in hash file generation by properly converting NumPy scalar types (bool, int, float) to Python native types.
Update to new SFINCS version.
Migrated documentation to mkdocs
Added a new option to detect floods based on actual discharge values from the hydrological model
Added a new option so that households can adapt to actual floods in the model
Implemented a simple version on runoff concentration so runoff is slowed down on its way to become discharge
Updated the performance_hydrodynamics function so it uses a list of observation files and matches these to the right flood map per event from sfincs. The name of the observation file has to be the same of the flood event (i.e. startdate - enddate.zarr)
For evaluation, the evaluation module now needs to be prefixed. So geb evaluate --methods plot_discharge becomes geb evaluate --methods hydrology.plot_discharge. Because we now have multiple evaluation files this keeps the logic clear.
dict in the input files in now updated to params. It is recommended to change the dict entry to params manually. Otherwise, it is also possible to re-build the model. This is because dict was giving issues with the type checker, because dict is a reserved name.

To support this version:

It is required to change the dict entry to params manually in input/files.yml. Otherwise, it is also possible to re-build the model.

Hydrosphere - Freshwater and Hydrology - Python
Published by jensdebruijn 7 months ago

What's Changed

Improve model startup time
Improve detection of outflow boundaries. Now uses intersection between river lines and geometry boundary.
Add an option in the config to run only coastal models.
Add tests for building a coastal model.
Many type fixes
Refactor reporter
By default export discharge data for outflow points
Use ZSTD compressor by default in write_zarr. This fixes a continuing error where forcing data was sometimes NaN
Use ZSTD compressor in reporter. This makes exporting data much faster.
Use a dynamically sized buffer to make writing in reporter more efficient, and reduce number of output files.
Remove annotations from docstrings in farmers.py
Do not use self in setup_donor_countries
Export discharge at outflow points by default (new setting in report: _outflow_points: true/false)
Add some tests for reporting
Remove support for Blosc encoding because of random async issues. Requires re-run of setup_forcing and setup_spei
Move examples to geb dir, so that they are included in the wheel

To support this version:

Re-run setup_forcing and setup_spei

Full Changelog: https://github.com/GEB-model/GEB/compare/v1.0.0b7...v1.0.0b8

Hydrosphere - Freshwater and Hydrology - Python
Published by jensdebruijn 8 months ago

GEB - v1.0.0b7

Hydrosphere - Freshwater and Hydrology - Python
Published by jensdebruijn 8 months ago

GEB - 1.0.0b6

Hydrosphere - Freshwater and Hydrology - Python
Published by jensdebruijn about 1 year ago

Fixes for study areas with no lakes and reservoirs
Fix issue with loading reasonable defaults config when geb was installed from PyPi
Allow exporting of several variables for water balance
Fix to allow exporting loading exported zarr files in xarray