{"id":306207,"name":"ChaosBench","description":"A benchmark project to improve and extend the predictability range of deep weather emulators to the subseasonal-to-seasonal (S2S) scale.","url":"https://github.com/leap-stc/chaosbench","last_synced_at":"2026-05-26T22:00:36.876Z","repository":{"id":220005454,"uuid":"688201217","full_name":"leap-stc/ChaosBench","owner":"leap-stc","description":null,"archived":false,"fork":false,"pushed_at":"2025-03-13T21:33:27.000Z","size":69963,"stargazers_count":52,"open_issues_count":2,"forks_count":7,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-05-22T21:49:06.209Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/leap-stc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-06T21:32:29.000Z","updated_at":"2026-05-22T13:59:48.000Z","dependencies_parsed_at":"2024-03-04T05:26:52.206Z","dependency_job_id":"15a3c24e-2336-4e38-a098-3e3156d1f158","html_url":"https://github.com/leap-stc/ChaosBench","commit_stats":{"total_commits":60,"total_committers":2,"mean_commits":30.0,"dds":0.01666666666666672,"last_synced_commit":"8bb356f236fe71a350944ea05943d340835fbef9"},"previous_names":["leap-stc/chaosbench"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/leap-stc/ChaosBench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leap-stc%2FChaosBench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leap-stc%2FChaosBench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leap-stc%2FChaosBench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leap-stc%2FChaosBench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/leap-stc","download_url":"https://codeload.github.com/leap-stc/ChaosBench/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leap-stc%2FChaosBench/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33450402,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-24T19:21:36.376Z","status":"ssl_error","status_checked_at":"2026-05-24T19:21:10.562Z","response_time":57,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"owner":{"login":"leap-stc","name":"LEAP","uuid":"90631797","kind":"organization","description":"Learning the Earth with AI and Physics","email":null,"website":null,"location":"United States of America","twitter":null,"company":null,"icon_url":"https://avatars.githubusercontent.com/u/90631797?v=4","repositories_count":2,"last_synced_at":"2023-03-09T08:25:24.412Z","metadata":{"has_sponsors_listing":false},"html_url":"https://github.com/leap-stc","funding_links":[],"total_stars":null,"followers":null,"following":null,"created_at":"2022-11-18T21:45:57.158Z","updated_at":"2023-03-09T08:25:24.450Z","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/leap-stc","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/leap-stc/repositories"},"packages":[],"commits":{"id":7920293,"full_name":"leap-stc/chaosbench","default_branch":"main","total_commits":64,"total_committers":2,"total_bot_commits":0,"total_bot_committers":0,"mean_commits":32.0,"dds":0.015625,"past_year_total_commits":0,"past_year_total_committers":0,"past_year_total_bot_commits":0,"past_year_total_bot_committers":0,"past_year_mean_commits":0.0,"past_year_dds":0.0,"last_synced_at":"2026-05-24T21:00:54.777Z","last_synced_commit":"4f82f402e8a1a53b0d03f1363b5c79d1b8d49fd1","created_at":"2024-12-23T00:12:12.054Z","updated_at":"2026-05-24T21:00:38.969Z","committers":[{"name":"juannat7","email":"nathanieljuan@gmail.com","login":"juannat7","count":63},{"name":"YONGQUAN-QU","email":"55875749+YONGQUAN-QU","login":"YONGQUAN-QU","count":1}],"past_year_committers":[],"commits_url":"https://commits.ecosyste.ms/api/v1/hosts/GitHub/repositories/leap-stc%2Fchaosbench/commits","host":{"name":"GitHub","url":"https://github.com","kind":"github","last_synced_at":"2026-05-26T00:00:10.885Z","repositories_count":6242363,"commits_count":883009441,"contributors_count":34924526,"owners_count":1157836,"icon_url":"https://github.com/github.png","host_url":"https://commits.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://commits.ecosyste.ms/api/v1/hosts/GitHub/repositories"}},"issues_stats":{"full_name":"leap-stc/chaosbench","html_url":"https://github.com/leap-stc/chaosbench","last_synced_at":"2026-05-20T19:01:01.688Z","status":"error","issues_count":2,"pull_requests_count":2,"avg_time_to_close_issue":null,"avg_time_to_close_pull_request":109.5,"issues_closed_count":0,"pull_requests_closed_count":2,"pull_request_authors_count":2,"issue_authors_count":2,"avg_comments_per_issue":1.5,"avg_comments_per_pull_request":0.0,"merged_pull_requests_count":1,"bot_issues_count":0,"bot_pull_requests_count":0,"past_year_issues_count":2,"past_year_pull_requests_count":1,"past_year_avg_time_to_close_issue":null,"past_year_avg_time_to_close_pull_request":193.0,"past_year_issues_closed_count":0,"past_year_pull_requests_closed_count":1,"past_year_pull_request_authors_count":1,"past_year_issue_authors_count":2,"past_year_avg_comments_per_issue":1.5,"past_year_avg_comments_per_pull_request":0.0,"past_year_bot_issues_count":0,"past_year_bot_pull_requests_count":0,"past_year_merged_pull_requests_count":0,"created_at":"2024-12-23T00:12:16.033Z","updated_at":"2026-05-20T19:01:01.688Z","repository_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/repositories/leap-stc%2Fchaosbench","issues_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/repositories/leap-stc%2Fchaosbench/issues","issue_labels_count":{},"pull_request_labels_count":{},"issue_author_associations_count":{"NONE":2},"pull_request_author_associations_count":{"NONE":2,"COLLABORATOR":1},"issue_authors":{"willzhang3":1,"SauryChen":1},"pull_request_authors":{"Cas-Dec":2,"yongquan-qu":1},"host":{"name":"GitHub","url":"https://github.com","kind":"github","last_synced_at":"2026-05-24T00:00:44.577Z","repositories_count":14691455,"issues_count":33987090,"pull_requests_count":111311429,"authors_count":11273030,"icon_url":"https://github.com/github.png","host_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/repositories","owners_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/owners","authors_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/authors"},"past_year_issue_labels_count":{},"past_year_pull_request_labels_count":{},"past_year_issue_author_associations_count":{"NONE":1},"past_year_pull_request_author_associations_count":{},"past_year_issue_authors":{"willzhang3":1},"past_year_pull_request_authors":{},"maintainers":[{"login":"yongquan-qu","count":1,"url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/authors/yongquan-qu"}],"active_maintainers":[]},"events":{"total":{"PullRequestEvent":2,"ForkEvent":6,"IssuesEvent":2,"WatchEvent":28,"IssueCommentEvent":2,"PushEvent":27},"last_year":{"ForkEvent":1,"IssuesEvent":1,"WatchEvent":7,"IssueCommentEvent":1}},"keywords":[],"dependencies":[{"ecosystem":"actions","filepath":".github/workflows/build-book.yaml","sha":null,"kind":"manifest","created_at":"2024-01-30T20:54:40.756Z","updated_at":"2024-01-30T20:54:40.756Z","repository_link":"https://github.com/leap-stc/ChaosBench/blob/main/.github/workflows/build-book.yaml","dependencies":[{"id":15929512006,"package_name":"actions/checkout","ecosystem":"actions","requirements":"v3","direct":true,"kind":"composite","optional":false},{"id":15929512007,"package_name":"andstor/file-existence-action","ecosystem":"actions","requirements":"v2","direct":true,"kind":"composite","optional":false},{"id":15929512008,"package_name":"CumulusDS/get-yaml-paths-action","ecosystem":"actions","requirements":"v1.0.2","direct":true,"kind":"composite","optional":false},{"id":15929512511,"package_name":"tj-actions/changed-files","ecosystem":"actions","requirements":"v36","direct":true,"kind":"composite","optional":false},{"id":15929512937,"package_name":"conda-incubator/setup-miniconda","ecosystem":"actions","requirements":"v2","direct":true,"kind":"composite","optional":false},{"id":15929513617,"package_name":"actions/cache","ecosystem":"actions","requirements":"v3","direct":true,"kind":"composite","optional":false},{"id":15929513618,"package_name":"actions/upload-artifact","ecosystem":"actions","requirements":"v4","direct":true,"kind":"composite","optional":false}]},{"ecosystem":"actions","filepath":".github/workflows/deploy-book.yaml","sha":null,"kind":"manifest","created_at":"2024-01-30T20:54:42.097Z","updated_at":"2024-01-30T20:54:42.097Z","repository_link":"https://github.com/leap-stc/ChaosBench/blob/main/.github/workflows/deploy-book.yaml","dependencies":[{"id":15929513619,"package_name":"actions/download-artifact","ecosystem":"actions","requirements":"v3","direct":true,"kind":"composite","optional":false},{"id":15929513620,"package_name":"dawidd6/action-download-artifact","ecosystem":"actions","requirements":"v2.27.0","direct":true,"kind":"composite","optional":false},{"id":15929513621,"package_name":"peaceiris/actions-gh-pages","ecosystem":"actions","requirements":"v3.9.3","direct":true,"kind":"composite","optional":false}]},{"ecosystem":"actions","filepath":".github/workflows/publish-website.yml","sha":null,"kind":"manifest","created_at":"2024-01-30T20:54:42.110Z","updated_at":"2024-01-30T20:54:42.110Z","repository_link":"https://github.com/leap-stc/ChaosBench/blob/main/.github/workflows/publish-website.yml","dependencies":[]},{"ecosystem":"pypi","filepath":"requirements.txt","sha":null,"kind":"manifest","created_at":"2024-03-04T05:26:50.480Z","updated_at":"2024-03-04T05:26:50.480Z","repository_link":"https://github.com/leap-stc/ChaosBench/blob/main/requirements.txt","dependencies":[{"id":16586540478,"package_name":"cdsapi","ecosystem":"pypi","requirements":"==0.6.1","direct":true,"kind":"runtime","optional":false},{"id":16586540479,"package_name":"cfgrib","ecosystem":"pypi","requirements":"==0.9.10.4","direct":true,"kind":"runtime","optional":false},{"id":16586540480,"package_name":"ecmwf_api_client","ecosystem":"pypi","requirements":"==1.6.3","direct":true,"kind":"runtime","optional":false},{"id":16586540481,"package_name":"einops","ecosystem":"pypi","requirements":"==0.7.0","direct":true,"kind":"runtime","optional":false},{"id":16586540482,"package_name":"lightning","ecosystem":"pypi","requirements":"==2.0.9","direct":true,"kind":"runtime","optional":false},{"id":16586540483,"package_name":"matplotlib","ecosystem":"pypi","requirements":"==3.7.2","direct":true,"kind":"runtime","optional":false},{"id":16586540484,"package_name":"numpy","ecosystem":"pypi","requirements":"==1.25.2","direct":true,"kind":"runtime","optional":false},{"id":16586540485,"package_name":"pandas","ecosystem":"pypi","requirements":"==2.0.3","direct":true,"kind":"runtime","optional":false},{"id":16586540486,"package_name":"PyYAML","ecosystem":"pypi","requirements":"==6.0","direct":true,"kind":"runtime","optional":false},{"id":16586540487,"package_name":"PyYAML","ecosystem":"pypi","requirements":"==6.0.1","direct":true,"kind":"runtime","optional":false},{"id":16586540488,"package_name":"scipy","ecosystem":"pypi","requirements":"==1.12.0","direct":true,"kind":"runtime","optional":false},{"id":16586540489,"package_name":"seaborn","ecosystem":"pypi","requirements":"==0.13.2","direct":true,"kind":"runtime","optional":false},{"id":16586540490,"package_name":"timm","ecosystem":"pypi","requirements":"==0.9.7","direct":true,"kind":"runtime","optional":false},{"id":16586540491,"package_name":"torch","ecosystem":"pypi","requirements":"==2.0.1","direct":true,"kind":"runtime","optional":false},{"id":16586540492,"package_name":"torchist","ecosystem":"pypi","requirements":"==0.2.1","direct":true,"kind":"runtime","optional":false},{"id":16586540493,"package_name":"tqdm","ecosystem":"pypi","requirements":"==4.66.1","direct":true,"kind":"runtime","optional":false},{"id":16586540494,"package_name":"xarray","ecosystem":"pypi","requirements":"==2023.8.0","direct":true,"kind":"runtime","optional":false}]},{"ecosystem":"pypi","filepath":"website/environment.yml","sha":null,"kind":"manifest","created_at":"2024-03-04T05:26:50.976Z","updated_at":"2024-03-04T05:26:50.976Z","repository_link":"https://github.com/leap-stc/ChaosBench/blob/main/website/environment.yml","dependencies":[]}],"score":4.68213122712422,"created_at":"2024-12-23T00:12:02.516Z","updated_at":"2026-05-26T22:00:36.877Z","avatar_url":"https://github.com/leap-stc.png","language":"HTML","category":"Climate Change","sub_category":"Earth and Climate Modeling","monthly_downloads":0,"total_dependent_repos":0,"total_dependent_packages":0,"readme":"# ChaosBench: A Multi-Channel, Physics-Based Benchmark for Subseasonal-to-Seasonal Climate Prediction\n\n\u003cdiv align=\"center\"\u003e\n  \u003ch3\u003eNeurIPS 2024 Oral\u003c/h3\u003e\n\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e\n\u003ca href=\"https://leap-stc.github.io/ChaosBench\"\u003e\u003cimg src=\"https://img.shields.io/badge/View-Documentation-blue?style=for-the-badge)\" alt=\"Homepage\"/\u003e\u003c/a\u003e\n  \u003ca href=\"https://arxiv.org/abs/2402.00712\"\u003e\u003cimg src=\"https://img.shields.io/badge/ArXiV-2402.00712-b31b1b.svg\" alt=\"arXiv\"/\u003e\u003c/a\u003e\n\u003ca href=\"https://huggingface.co/datasets/LEAP/ChaosBench\"\u003e\u003cimg src=\"https://img.shields.io/badge/Dataset-HuggingFace-ffd21e\" alt=\"Huggingface Dataset\"/\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/leap-stc/ChaosBench/blob/main/LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/badge/License-GNU%20GPL-green\" alt=\"License Badge\"/\u003e\u003c/a\u003e\n\u003c/div\u003e\n\u003c/br\u003e\n\nChaosBench is a benchmark project to improve and extend the predictability range of deep weather emulators to the subseasonal-to-seasonal (S2S) range. Predictability at this scale is more challenging due to its: (1) __double sensitivities__ to intial condition (in weather-scale) and boundary condition (in climate-scale), (2) __butterfly effect__, and our (3) __inherent lack of understanding__ of physical processes operating at this scale. Thus, given the __high socioeconomic stakes__ for accurate, reliable, and stable S2S forecasts (e.g., for disaster/extremes preparedness), this benchmark is timely for DL-accelerated solutions.\n\n\u003e 💡 View our interactive __leaderboard__ [here](https://leap-stc.github.io/ChaosBench/leaderboard.html)\n\n\n## Features\n\n![Overview of ChaosBench](docs/scheme/chaosbench_scheme-scheme.jpg)\n\n1️⃣ __Diverse Observations__. Spanning over 45 years (1979-), we include ERA5/LRA5/ORAS5 reanalysis for a fully-coupled Earth system emulation (atmosphere-terrestrial-sea-ice)\n\n2️⃣ __Diverse Baselines__. Wide selection of physics-based forecasts from leading national weather agencies in Europe, the UK, America, and Asia\n\n3️⃣ __Differentiable Physics Metrics__. In addition to deterministic and probabilistic metrics, we introduce two differentiable physics-based metrics to minimize the decay of power spectra at long forecasting horizon (reduce blurriness)\n\n4️⃣ __Large-Scale Benchmarking__. Systematic large-scale evaluation for state-of-the-art ML-based weather emulators like ViT/ClimaX, PanguWeather, GraphCast, and FourcastNetV2\n\n## Getting Started\n- [Motivation](https://leap-stc.github.io/ChaosBench/motivation.html)\n- [Quickstart](https://leap-stc.github.io/ChaosBench/quickstart.html)\n- [Dataset Overview](https://leap-stc.github.io/ChaosBench/dataset.html)\n\n\n## Build Your Own Model\n- [Training](https://leap-stc.github.io/ChaosBench/training.html)\n- [Evaluation](https://leap-stc.github.io/ChaosBench/evaluation.html)\n\n## Benchmarking\n- [Baseline Models](https://leap-stc.github.io/ChaosBench/baseline.html)\n- [Leaderboard](https://leap-stc.github.io/ChaosBench/leaderboard.html)\n\n## Citation\nIf you find any of the code and dataset useful, feel free to acknowledge our work through:\n\n```bibtex\n@article{nathaniel2024chaosbench,\n  title={Chaosbench: A multi-channel, physics-based benchmark for subseasonal-to-seasonal climate prediction},\n  author={Nathaniel, Juan and Qu, Yongquan and Nguyen, Tung and Yu, Sungduk and Busecke, Julius and Grover, Aditya and Gentine, Pierre},\n  journal={arXiv preprint arXiv:2402.00712},\n  year={2024}\n}\n\n","funding_links":[],"readme_doi_urls":[],"works":{},"citation_counts":{},"total_citations":0,"keywords_from_contributors":[],"project_url":"https://ost.ecosyste.ms/api/v1/projects/306207","html_url":"https://ost.ecosyste.ms/projects/306207"}