{"id":349139,"name":"Top Of The Poops","description":"Analysing Sewage Information from the UK Environment Agency.","url":"https://github.com/top-poop/top-of-the-poops","last_synced_at":"2026-06-13T05:00:35.208Z","repository":{"id":160487118,"uuid":"422171447","full_name":"top-poop/top-of-the-poops","owner":"top-poop","description":"Analysing Sewage Information from the UK Environment Agency","archived":false,"fork":false,"pushed_at":"2026-03-30T08:17:39.000Z","size":14212,"stargazers_count":26,"open_issues_count":10,"forks_count":6,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-06-05T01:03:59.798Z","etag":null,"topics":["environment-agency","sewage","uk-government"],"latest_commit_sha":null,"homepage":"https://top-of-the-poops.org","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/top-poop.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2021-10-28T11:07:20.000Z","updated_at":"2026-04-18T22:43:02.000Z","dependencies_parsed_at":null,"dependency_job_id":"14940c7b-f08a-44a2-92cf-9d4fd9782c7d","html_url":"https://github.com/top-poop/top-of-the-poops","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/top-poop/top-of-the-poops","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/top-poop%2Ftop-of-the-poops","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/top-poop%2Ftop-of-the-poops/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/top-poop%2Ftop-of-the-poops/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/top-poop%2Ftop-of-the-poops/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/top-poop","download_url":"https://codeload.github.com/top-poop/top-of-the-poops/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/top-poop%2Ftop-of-the-poops/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34272603,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-13T02:00:06.617Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"owner":{"login":"top-poop","name":"Top Poop","uuid":"93323279","kind":"user","description":"","email":"","website":null,"location":null,"twitter":null,"company":null,"icon_url":"https://avatars.githubusercontent.com/u/93323279?u=33b53dae1cbf201b9d9b8dd1d4fd30eeb2fa73f4\u0026v=4","repositories_count":2,"last_synced_at":"2024-05-01T10:58:58.063Z","metadata":{"has_sponsors_listing":true},"html_url":"https://github.com/top-poop","funding_links":["https://github.com/sponsors/top-poop"],"total_stars":13,"followers":6,"following":0,"created_at":"2023-05-02T15:36:24.739Z","updated_at":"2024-05-01T10:58:59.837Z","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/top-poop","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/top-poop/repositories"},"packages":[],"commits":{"id":11690419,"full_name":"top-poop/top-of-the-poops","default_branch":"master","total_commits":279,"total_committers":1,"total_bot_commits":0,"total_bot_committers":0,"mean_commits":279.0,"dds":0.0,"past_year_total_commits":10,"past_year_total_committers":1,"past_year_total_bot_commits":0,"past_year_total_bot_committers":0,"past_year_mean_commits":10.0,"past_year_dds":0.0,"last_synced_at":"2026-06-11T04:11:51.368Z","last_synced_commit":"cddf9bd025e2e5cdae849baaeda06cae601b51da","created_at":"2026-03-21T12:40:26.097Z","updated_at":"2026-06-11T04:11:39.743Z","committers":[{"name":"Top Poop","email":"93323279+top-poop","login":"top-poop","count":279}],"past_year_committers":[{"name":"Top Poop","email":"93323279+top-poop","login":"top-poop","count":10}],"commits_url":"https://commits.ecosyste.ms/api/v1/hosts/GitHub/repositories/top-poop%2Ftop-of-the-poops/commits","host":{"name":"GitHub","url":"https://github.com","kind":"github","last_synced_at":"2026-06-13T00:00:12.063Z","repositories_count":6257917,"commits_count":882318665,"contributors_count":35075763,"owners_count":1165806,"icon_url":"https://github.com/github.png","host_url":"https://commits.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://commits.ecosyste.ms/api/v1/hosts/GitHub/repositories"}},"issues_stats":{"full_name":"top-poop/top-of-the-poops","html_url":"https://github.com/top-poop/top-of-the-poops","last_synced_at":"2026-05-21T18:01:52.197Z","status":"error","issues_count":20,"pull_requests_count":3,"avg_time_to_close_issue":33237704.0,"avg_time_to_close_pull_request":12724463.0,"issues_closed_count":12,"pull_requests_closed_count":1,"pull_request_authors_count":2,"issue_authors_count":19,"avg_comments_per_issue":2.1,"avg_comments_per_pull_request":1.3333333333333333,"merged_pull_requests_count":0,"bot_issues_count":0,"bot_pull_requests_count":2,"past_year_issues_count":2,"past_year_pull_requests_count":0,"past_year_avg_time_to_close_issue":8586277.0,"past_year_avg_time_to_close_pull_request":null,"past_year_issues_closed_count":2,"past_year_pull_requests_closed_count":0,"past_year_pull_request_authors_count":0,"past_year_issue_authors_count":2,"past_year_avg_comments_per_issue":3.5,"past_year_avg_comments_per_pull_request":null,"past_year_bot_issues_count":0,"past_year_bot_pull_requests_count":0,"past_year_merged_pull_requests_count":0,"created_at":"2025-08-29T23:17:36.270Z","updated_at":"2026-05-21T18:01:52.197Z","repository_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/repositories/top-poop%2Ftop-of-the-poops","issues_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/repositories/top-poop%2Ftop-of-the-poops/issues","issue_labels_count":{"data-sources":1},"pull_request_labels_count":{"dependencies":2},"issue_author_associations_count":{"NONE":20},"pull_request_author_associations_count":{"NONE":3},"issue_authors":{"V2G-EVSE":2,"KJ-UoL":1,"jonese1":1,"AmySlack":1,"seagulljim":1,"cleansouthernwater":1,"FelixAJNobes":1,"asibs":1,"paul-hammant":1,"petercrwilliams2013-jpg":1,"100Nicola":1,"browndg":1,"Stekkles":1,"MadMackMcMac":1,"Darkdeap":1,"VickiElcoate":1,"euoia":1,"VGL-LGV":1,"jahjones":1},"pull_request_authors":{"dependabot[bot]":2,"asibs":1},"host":{"name":"GitHub","url":"https://github.com","kind":"github","last_synced_at":"2026-06-09T00:00:17.805Z","repositories_count":14785464,"issues_count":33468540,"pull_requests_count":110234754,"authors_count":11296598,"icon_url":"https://github.com/github.png","host_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/repositories","owners_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/owners","authors_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/authors"},"past_year_issue_labels_count":{},"past_year_pull_request_labels_count":{},"past_year_issue_author_associations_count":{"NONE":2},"past_year_pull_request_author_associations_count":{},"past_year_issue_authors":{"euoia":1,"petercrwilliams2013-jpg":1},"past_year_pull_request_authors":{},"maintainers":[],"active_maintainers":[]},"events":{"total":{"ForkEvent":1,"IssuesEvent":1,"WatchEvent":5,"IssueCommentEvent":1,"PushEvent":5},"last_year":{"IssuesEvent":1,"WatchEvent":1,"IssueCommentEvent":1,"PushEvent":4}},"keywords":["environment-agency","sewage","uk-government"],"dependencies":[],"score":3.58351893845611,"created_at":"2026-03-21T12:40:26.874Z","updated_at":"2026-06-13T05:00:35.209Z","avatar_url":"https://github.com/top-poop.png","language":"Python","category":"Natural Resources","sub_category":"Water Supply and Quality","monthly_downloads":0,"total_dependent_repos":0,"total_dependent_packages":0,"readme":"\n\n# Top Of The Poops\n\nWebsite: [top-of-the-poops.org](https://top-of-the-poops.org)\n\nSeems that [#sewage](https://twitter.com/search?q=%23sewage) is on people's minds right now.\n\nThe UK publishes some information about sewage outfalls - here are some scripts to get this information, analyse it, and\nperhaps publish some interesting findings.\n\n## Data Reuse and Attribution\n\nPlease re-use our data.\n\nPress contact: press [at] top-of-the-poops.org\n\nIf you publish data, content, or images from our site, please note it is [CC-BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/), and as such we require suitable *attribution*\n\n- Derived Data / General Content - should be attributed, with name and hyperlink  [Top of the Poops](https://top-of-the-poops.org)\n- Images / Maps - should have caption '(c) top-of-the-poops.org', or similar, either as plain text or hyperlink, and ther should be a hyperlink as above in the main body of the text.\n\nPlease refer to: https://wiki.creativecommons.org/wiki/Recommended_practices_for_attribution\n\nDerived data is (C) Top-Of-The-Poops - [CC-BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/), all original data is (C) the original data owner, and is used under appropriate licence \n\n\n## Maps\n\nWe previously used mapbox but after getting very popular we couldn't afford it anymore!\nMaps now rendered ourselves, but it's not going to be as fast as MapBox.\n\nWe use [TileServer GL](https://github.com/maptiler/tileserver-gl) in combination with a UK Vector map\nfrom [MapTiler](https://www.maptiler.com/data/)\n\n\n## How to use\n\nYou can clone the repo - I use IntelliJ IDEA to make a hot-reloading web page. \nThe build runs locally with `make watch`\n\nAll data files are generated on developer machine, only the javascript build runs on CI. This ensures the CI build is acceptably fast.\nCurrently it runs in about 10 seconds. Which is OK, could be faster.\n\n`make watch` uses inotify - this may not work on MacOS.\n\n## Contributing\n\nContributions are welcome - especially CSS / Javascript improvements! But please chat before doing any real work - to make sure everyone is aligned with direction. \n\n### Development Environment\n\nThis has been developed on Linux, the makefiles may or may not work on a Mac.\n\n### Setting up the database\n\n```shell\nmake python\ncd db/data\nmake load-all\n```\n\n### Generating json data files\n\nYou'll need to have set up the database stuff first\n\n```shell\nmake generated\n```\n\n\n### React\n\nWhy is there a React app per page? \nBecause it makes it easy to write the software\n\n### MP Data \n\nhttps://www.theyworkforyou.com/mps/?f=csv\nhttps://www.politics-social.com/list/name\n\n#### Not fetched yet\n\nhttp://everypolitician.org/uk/commons/download.html\n\nhttps://www.ukinbound.org/wp-content/uploads/2020/07/List-of-MPs-with-active-Twitter-accounts-organised-by.pdf\n\n\n### Constituency Shapes\n\nhttps://opendata.arcgis.com/api/v3/datasets/19841da5f8f6403e9fdcfb35c16e11e9_0/downloads/data?format=shp\u0026spatialRefId=27700\n\nSource: Office for National Statistics licensed under the Open Government Licence v.3.0 \n\nContains OS data © Crown copyright and database right 2021\n\n### Sewage Data\n\nEvent Duration Monitoring\n\nhttps://environment.data.gov.uk/dataset/21e15f12-0df8-4bfc-b763-45226c16a8ac\nhttps://environment.data.gov.uk/portalstg/home/item.html?id=045af51b3be545b79b0c219811d3d243\nhttps://environment.data.gov.uk/portalstg/sharing/rest/content/items/045af51b3be545b79b0c219811d3d243/data\n\n# 2022\n\nhttps://environment.data.gov.uk/portalstg/home/item.html?id=2f8d9b7628dd4f60a30fb1a8483fc2ae\n\nConsented Discharges with Conditions\n\nhttps://environment.data.gov.uk/dataset/5fe5ab2e-d465-11e4-8a42-f0def148f590\nhttps://environment.data.gov.uk/portalstg/sharing/rest/content/items/5e618f2b5c7f47cca44eb468aa2e43f0/data\n\n#### Wales\nConsented Discharges with Conditions\n\nhttps://lle.gov.wales/catalogue/item/ConsentedDischargesToControlledWatersWithConditions/?lang=en\nhttps://naturalresourceswales.sharefile.eu/share/view/s05adea6ab5d4df58/fo289e69-abc0-4acb-9923-271512440118\nhttps://storage-eu-205.sharefile.com/download.ashx?dt=dt99e5eec3bd194293acd60049575d41ee\u0026cid=9AQXBd2ldhvlRrRbQ8tE-w\u0026zoneid=zpc3159d90-01f7-41a7-a8ab-3704157466\u0026exp=1637152468\u0026zsid=FB\u0026h=F%2BC3TQBtcWx%2BYjb4jglnxmRAZLWwiRKrwDw7xn%2BoShI%3D\n\n\nEvent Duration Monitoring\n\n2020 - Can't find! - Partial information at: https://www.dwrcymru.com/en/our-services/wastewater/combined-storm-overflows/valleys-and-south-east-wales\n\n2021 - Main page: https://www.dwrcymru.com/en/our-services/wastewater/river-water-quality/combined-storm-overflows\n2021 - Seems to be split over 3 files (with different formats), unknown overlap with Environment Agency data.\n - https://www.dwrcymru.com/-/media/Project/Files/Page-Documents/Our-Services/Wastewater/CSO/EDM-Return-Dwr-Cymru-Welsh-Water-Emergency-Overflow-Annual-2021.ashx\n - https://www.dwrcymru.com/-/media/Project/Files/Page-Documents/Our-Services/Wastewater/CSO/EDM-Return-Dwr-Cymru-Welsh-Water-Storm-Overflow-Annual-2021.ashx\n - https://www.dwrcymru.com/-/media/Project/Files/Page-Documents/Our-Services/Wastewater/CSO/EDM-Return-DCWW_Wales-Water-Annual-2021.ashx\n\n\n# Bathing \n\nBathing Water Monitoring Locations\nhttps://www.data.gov.uk/dataset/dcb8bd46-c4cf-4749-bad0-7663da96845c/bathing-waters-monitoring-locations\n Name + Classification by year\n\nSensitive Areas Bathing\nhttps://www.data.gov.uk/dataset/4e2bbdb4-15d3-49dc-ba22-904045b091fb/sensitive-areas-bathing-waters\nhttps://datamap.gov.wales/layers/inspire-nrw:NRW_UWWTD_SA_BATHING_WATERS\n\n\n\n\n\n### Postcodes\n\nhttps://geoportal.statistics.gov.uk/datasets/ons-postcode-directory-february-2020/about\n\nhttps://data.gov.uk/dataset/6de48d19-b3a0-4e45-b98e-01bd781b035c/ons-postcode-directory-latest-centroids\n\nhttp://geoportal1-ons.opendata.arcgis.com/datasets/75edec484c5d49bcadd4893c0ebca0ff_0.csv?outSR={%22latestWkid%22:27700,%22wkid%22:27700}\n\n### Software\n\nYou'll need the following:\n\n- python3\n- libreoffice\n- gdal-bin\n\n# Things to do\n\n- Link with voting results - need to find the division results...\n- Rivers and beaches by constituency?\n- Constituency page showing all the things by constituency?\n\n\n# Data Quality\n\nTo be sure the quality of the data is unbelievably poor. Perhaps it is so poor so that it is hard to understand?\n\n## 2021 Data\n\n### Issues\n - Distributed as an Excel file, which is hard to process\n   - Should ideally be a machine readable format. I'll say simple XML, with schema, but a consistent CSV file would be OK.\n - Mix and match of data types \n   - Numeric columns have \"N/A\", \"#N/A\", and \"#NA\"\n   - Name columns have \"0\" in\n   - Percentage values scale from 0-1 in some sheets, and 0-100 in others, because some sheets have cells set to \"Numeric\", and others to \"Percent\"\n - Continuation rows\n   - A few of the sheets don't stick to \"one row per record\", which is kinda mandatory in a machine readable file.\n - Inconsistent data\n   - Particularly consent ids don't match consent ids in the consent database - the formatting differs\n   - Consent ids don't have a consistent format.\n   - Loads of EDM rows don't match valid consents. \n - Duplicate data rows \n   - Some data rows are duplicates in many of the source files. It is not clear why, it looks like an extract from a database upstream has maybe repeated rows where there are multiple assets with the same consent information?\n - Wales data is spread over multiple files, with different formats, and may or may not overlap with EA data.\n\n\n### Noted Improvements\n - The files are now tabs in a single document with *almost* consistent data across the tabs.\n\n### Example Duplicate Data\n\n```\n'Anglian Water', 'DAVENTRY SEWER SYSTEM', 'AW5NF181', 'A1'\n'Dwr Cymru Welsh Water', '#TBC', '#N/A', '', '', '', '0.25', '1', '100', ''\n'Severn Trent Water', 'WITTON - GEORGE ROAD XXX (CSO)', 'TBC', '', '', '', '', '', '', ''\n'South West Water', 'KINGSAND SEWAGE PUMPING STATION', '301903', 'A1', '', 'KINGSAND BEACH', '33.72', '21', '100', ''\n```\n\n","funding_links":[],"readme_doi_urls":[],"works":{},"citation_counts":{},"total_citations":0,"keywords_from_contributors":[],"project_url":"https://ost.ecosyste.ms/api/v1/projects/349139","html_url":"https://ost.ecosyste.ms/projects/349139"}