WorldCereal - WorldCereal classification module v2.4.1

What's Changed

This release makes the long-awaited transition in the background from the non-flexible presto-worldcereal library to the prometheo library (https://github.com/WorldCereal/prometheo) which is future proof in terms of pretrained geospatial models that are supported. This allows WorldCereal in the future to onboard other models like Presto with limited code changes.

This release also simplifies the openEO-based workflows (UDFs) for further improvement gains (and more to come!).

From this release onward, the "focus token" approach is replaced by a "temporal-explicit" prediction and "target-date" based selection of the timestamp of interest. More information in the docs section of the repo.

Many smaller updates are part of this release as well. Detailed change commits are listed below.

Detailed changes

• timestamps computation update by @giollimirgia in https://github.com/WorldCereal/worldcereal-classification/pull/358
• Notebook updates by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/368
• Cropland notebook by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/369
• Updated test resource by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/370
• Update production grid by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/371
• pass augment argument by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/372
• #347-improve-patch-to-point by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/348
• Fix tz awareness point extractions by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/374
• Add B8A to inference by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/376
• make cropland masking optional during crop type inference by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/380
• Add B8A to inference by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/381
• Temporal Extent Alignment by @cbutsko in https://github.com/WorldCereal/worldcereal-classification/pull/383
• Notebook improvements by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/387
• Fail post-job-action when dimensions don't match by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/390
• Bump to scikit-learn=1.6.1 by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/391
• Check STAC for existing extractions by @cbutsko in https://github.com/WorldCereal/worldcereal-classification/pull/384
• Disable S1 in the patch-to-point process graph in case no extractions are available by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/393
• Remove GFMap Dependencies in inference UDFs by @VincentVerelst in https://github.com/WorldCereal/worldcereal-classification/pull/392
• Switch to prometheo by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/269
• Bump gfmap version to 0.4.7 by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/412

New Contributors

• @giollimirgia made their first contribution in https://github.com/WorldCereal/worldcereal-classification/pull/358

Full Changelog: https://github.com/WorldCereal/worldcereal-classification/compare/v2.3.0...v2.4.1

Consumption - Agriculture and Nutrition - Jupyter Notebook
Published by kvantricht 10 months ago

WorldCereal - WorldCereal classification module v2.3.0

What's Changed

As most important change for this release, we now added support for large-scale inference by using openEO's Multi Backend Job Manager. We now also support using a shapefile or geopackage as spatial extent specifier in the demo notebooks.

Specific changes:

• Run inference workflow on Python 3.11 image of openeo by @VincentVerelst in https://github.com/WorldCereal/worldcereal-classification/pull/362
• new script demonstrating the use of croptype udp by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/360
• map ui also supports shp and gpkg upload by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/366
• Correct rdm notebook by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/365
• Added large-scale inference handler by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/367

Full Changelog: https://github.com/WorldCereal/worldcereal-classification/compare/v2.2.3...v2.3.0

Consumption - Agriculture and Nutrition - Jupyter Notebook
Published by kvantricht 12 months ago

WorldCereal - WorldCereal classification module v2.2.3

What's Changed

• Fix typos in documentation by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/355
• Fix collect_inputs job options usage by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/354
• Fix DOY calculation and S3URI message by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/353
• added checks on empty extraction dfs by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/357
• avoid duckdb syntax error when multiple private collections by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/356
• 327 lut crop type by @VincentVerelst in https://github.com/WorldCereal/worldcereal-classification/pull/346

Full Changelog: https://github.com/WorldCereal/worldcereal-classification/compare/v2.2.2...v2.2.3

Consumption - Agriculture and Nutrition - Jupyter Notebook
Published by kvantricht about 1 year ago

WorldCereal - WorldCereal classification module v2.2.2

What's Changed

• updated S2 stac metadata #329 by @VincentVerelst in https://github.com/WorldCereal/worldcereal-classification/pull/330
• Patch-extraction-improvements by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/328
• #308 add minimal pipeline test by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/309
• Added a create_patch_to_point_worldcereal function by @VincentVerelst in https://github.com/WorldCereal/worldcereal-classification/pull/316
• Update query extractions by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/334
• ensure irrigation_status included when retrieving samples by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/335
• consistency in default RDM attributes by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/336
• Require openeo-gfmap v0.4.4 by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/337
• Avoid global import of grp module by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/343
• Better handling of spatial filtering by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/332
• Make compositing window flexible by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/349
• Remove combined use of save_result and out_format in execute_batch by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/351

Full Changelog: https://github.com/WorldCereal/worldcereal-classification/compare/v2.2.0...v2.2.2

Consumption - Agriculture and Nutrition - Jupyter Notebook
Published by kvantricht about 1 year ago

WorldCereal - WorldCereal classification module v2.2.0

This v2.2.0 release of the system includes numerous updates and fixes.

What's Changed

• #280 Use new monthly composited meteo collection by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/281
• Update patch post job actions by @VincentVerelst in https://github.com/WorldCereal/worldcereal-classification/pull/278
• added worldcereal_crop_extent and type UDP to the repo by @VincentVerelst in https://github.com/WorldCereal/worldcereal-classification/pull/284
• connect to CDSE with correct URL by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/289
• demonstration on how to use cropland extent udp by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/238
• Merge geoparquets after point extraction workflow by @VincentVerelst in https://github.com/WorldCereal/worldcereal-classification/pull/291
• allow visualization of multiple collections on the map by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/297
• fix private collection download by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/296
• move extraction functions scripts > src by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/292
• Support BaseGeometry as input for RdmInteraction.get_samples() by @VincentVerelst in https://github.com/WorldCereal/worldcereal-classification/pull/298
• Notebook v2 pt1 by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/299
• separate retrieval and plotting of calendars in diff functions by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/302
• Update UI map by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/301
• Update URL for agera5 precomposited catalogue by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/303
• Update process_parquet and querying of public/private extractions by @cbutsko in https://github.com/WorldCereal/worldcereal-classification/pull/300
• Def crop noteb by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/304
• Croptype noteb by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/306
• improve notebook documentation by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/310
• More efficient parquet loading by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/314
• Improvements for (patch) extractions by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/315
• Assume ref_id specific root_folder by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/317
• Patch extractions have to go into ref_id folder by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/318
• ensure consistent order of probability bands by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/322
• #319 flexible organization for extractions by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/320
• rdm auth now through CDSE by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/323
• Small patch extraction improvements by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/321

Full Changelog: https://github.com/WorldCereal/worldcereal-classification/compare/v2.1.2...v2.2.0

Consumption - Agriculture and Nutrition - Jupyter Notebook
Published by kvantricht about 1 year ago

WorldCereal - WorldCereal classification module v2.1.2

This v2.1.2 release of the system includes numerous updates and fixes. Most important addition in this version is the interaction with the RDM API to be able to query public and private reference data samples. This functionality is demonstrated in a RDM demo notebook.

What's Changed

• Catch an empty df and throw error by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/207
• avoid error if only one class present in result by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/208
• First version of a rdm api interaction module #125 by @VincentVerelst in https://github.com/WorldCereal/worldcereal-classification/pull/161
• Enable jenkins pre-install script by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/209
• tile size must now be multiple of 16 in openeo by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/218
• Consolidate extraction endpoints by @VincentVerelst in https://github.com/WorldCereal/worldcereal-classification/pull/216
• visualize training dataframe by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/224
• Rdm data extractions by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/227
• Write patch extraction metadata directly to STAC API by @VincentVerelst in https://github.com/WorldCereal/worldcereal-classification/pull/228
• update date slider by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/236
• introduce legend functionality + example script by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/237
• 234-refactor-point-extractions by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/235
• 245 fix querying public extraction function by @cbutsko in https://github.com/WorldCereal/worldcereal-classification/pull/246
• add slope band to post_job_action by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/249
• get_legend reads directly from URL + extended to IRR by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/254
• #233 allow setting minimum quality score when requesting samples from… by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/260
• Create extract module by @VincentVerelst in https://github.com/WorldCereal/worldcereal-classification/pull/271
• Rdm demo by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/274
• Do not use spatial_extent for load_stac by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/275
• updated rdm links by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/276

Full Changelog: https://github.com/WorldCereal/worldcereal-classification/compare/v2.1.0...v2.1.2

Consumption - Agriculture and Nutrition - Jupyter Notebook
Published by kvantricht over 1 year ago

WorldCereal - WorldCereal classification module v2.1.0

This is the first release of the completely refactored WorldCereal system!

What's Changed

• Stripped version for new design based on OpenEO by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/20
• #17 setup Github actions workflow by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/21
• #17 Setup Github actions by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/22
• Extraction pipelines & data fetching blocks by @GriffinBabe in https://github.com/WorldCereal/worldcereal-classification/pull/36
• Fixed long UDF by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/46
• Using the sample_id for items ID in the stac metadata by @GriffinBabe in https://github.com/WorldCereal/worldcereal-classification/pull/45
• Attempt to fix jenkins build by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/53
• MVP by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/47
• 44 translate cropno crop inference udf to gfmap by @GriffinBabe in https://github.com/WorldCereal/worldcereal-classification/pull/48
• Make batch_size for Presto flexible by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/59
• 56 meteo inference by @GriffinBabe in https://github.com/WorldCereal/worldcereal-classification/pull/60
• Seasons by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/66
• Now writing GeoTiff and having two bands: binary mask and probabilities by @GriffinBabe in https://github.com/WorldCereal/worldcereal-classification/pull/65
• Bugfix seasonality check and max season diff check by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/68
• Centralize worldcereal generate_map method by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/67
• PR 67 fixes by @GriffinBabe in https://github.com/WorldCereal/worldcereal-classification/pull/70
• Make version static for now by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/72
• Add openeo-gfmap dependency by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/74
• 57 udf memory profiling by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/77
• 71 generate map func by @GriffinBabe in https://github.com/WorldCereal/worldcereal-classification/pull/78
• Add croptype inference by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/75
• Define some auth related env vars for Jenkins by @soxofaan in https://github.com/WorldCereal/worldcereal-classification/pull/80
• batch_size as parameter by @jdries in https://github.com/WorldCereal/worldcereal-classification/pull/81
• Point extractions by @VincentVerelst in https://github.com/WorldCereal/worldcereal-classification/pull/84
• Update crop type mapping workflow by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/85
• Refactored fix by @GriffinBabe in https://github.com/WorldCereal/worldcereal-classification/pull/89
• Added configuration data classes for cropland/croptype mapping workflow by @GriffinBabe in https://github.com/WorldCereal/worldcereal-classification/pull/91
• Fix croptype/cropland parameters for pydantic>2 by @GriffinBabe in https://github.com/WorldCereal/worldcereal-classification/pull/94
• Infer classes from model by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/96
• Filter on S1 polarization by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/97
• Updated formatting-checks by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/104
• Formatting checks by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/105
• Custom tile size by @VincentVerelst in https://github.com/WorldCereal/worldcereal-classification/pull/106
• System V1 demo by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/95
• removed dependency on extract attribute for point extractions by @VincentVerelst in https://github.com/WorldCereal/worldcereal-classification/pull/107
• 79 s1 extractions by @GriffinBabe in https://github.com/WorldCereal/worldcereal-classification/pull/113
• Small fix by @GriffinBabe in https://github.com/WorldCereal/worldcereal-classification/pull/114
• Changed spatial resolution to 20m for S1 and small arg fix by @GriffinBabe in https://github.com/WorldCereal/worldcereal-classification/pull/126
• fixed typo in the s1 item_asset definitions in extract_sar.py by @VincentVerelst in https://github.com/WorldCereal/worldcereal-classification/pull/123
• #129 Add custom job_options by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/130
• 121 add slope & Improvement of S1 extractions by @GriffinBabe in https://github.com/WorldCereal/worldcereal-classification/pull/128
• #55 Immediately map to presto band names by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/131
• Renamed altitude->elevation as required by presto by @GriffinBabe in https://github.com/WorldCereal/worldcereal-classification/pull/132
• Compute slope with scipy, avoiding the use of richdem by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/138
• Bugfix dimension order by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/140
• Presto-compilation by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/133
• Add WorldCereal input extractions worfklow by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/137
• extract from first day to last day of month by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/148
• Enable mypy -> this is gonna take some effort by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/147
• Update preprocessing by @GriffinBabe in https://github.com/WorldCereal/worldcereal-classification/pull/149
• Enable pre_test_script.sh for jenkins by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/150
• Fix CI with openEO connections by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/152
• Add openEO tests by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/153
• Use gfmap-based backend connection by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/154
• 139-slope-collection by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/155
• Rebuild stac scripts by @GriffinBabe in https://github.com/WorldCereal/worldcereal-classification/pull/158
• Adding postprocessing to WorldCereal jobs by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/157
• Made the extractor main's loop robust to artifactory errors by @GriffinBabe in https://github.com/WorldCereal/worldcereal-classification/pull/162
• Updated crop calendars by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/173
• allow multiple outputs for inference results by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/175
• Postproc by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/177
• Refactoring of System V1 demo by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/170
• Allow custom job_options for collect_inputs by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/178
• Unified CropClassifier and no more binary postprocessing by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/180
• Fix crop/no-crop lookup table by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/181
• Speedup extractions from public Parquet by @cbutsko in https://github.com/WorldCereal/worldcereal-classification/pull/183
• Fix filenaming by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/185
• Specify which product is being generated by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/186
• Bugfix: do not assume a label order during postprocessing by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/187
• get rid of conf_threshold parameter as it may introduce "undecided" p… by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/188
• Make the use of class weights optional (default=False) by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/189
• Updated cropland model v006 by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/192
• Updated README for release by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/193
• Updated crop calendars by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/196
• cleaned notebooks for demo by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/190
• f!x call to postprocessparameters by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/197
• Fix UI map by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/198
• 194-align-training-window-with-user-defined-temporal-extent by @cbutsko in https://github.com/WorldCereal/worldcereal-classification/pull/195
• 179-add-hierarchical-croptype-picker by @cbutsko in https://github.com/WorldCereal/worldcereal-classification/pull/191
• Override parent method to suppress warning by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/199
• Add masking to custom model training by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/200
• Add basic visualization by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/201
• replaced warning with error and helper message by @cbutsko in https://github.com/WorldCereal/worldcereal-classification/pull/205
• Update notebooks by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/203
• added figures on data availability by @jdegerickx in https://github.com/WorldCereal/worldcereal-classification/pull/206
• Bump to 2.1.0 by @kvantricht in https://github.com/WorldCereal/worldcereal-classification/pull/202

New Contributors

• @kvantricht made their first contribution in https://github.com/WorldCereal/worldcereal-classification/pull/20
• @GriffinBabe made their first contribution in https://github.com/WorldCereal/worldcereal-classification/pull/36
• @soxofaan made their first contribution in https://github.com/WorldCereal/worldcereal-classification/pull/80
• @jdries made their first contribution in https://github.com/WorldCereal/worldcereal-classification/pull/81
• @VincentVerelst made their first contribution in https://github.com/WorldCereal/worldcereal-classification/pull/84
• @jdegerickx made their first contribution in https://github.com/WorldCereal/worldcereal-classification/pull/175
• @cbutsko made their first contribution in https://github.com/WorldCereal/worldcereal-classification/pull/183

Full Changelog: https://github.com/WorldCereal/worldcereal-classification/compare/v1.1.1...v2.1.0

Consumption - Agriculture and Nutrition - Jupyter Notebook
Published by kvantricht over 1 year ago

WorldCereal - WorldCereal classification module v1.1.1

What's changed

This is the first public release of the WorldCereal classification module!

Consumption - Agriculture and Nutrition - Jupyter Notebook
Published by kvantricht about 3 years ago