{"id":325194,"name":"reformatters","description":"Reformat weather datasets into Zarr.","url":"https://github.com/dynamical-org/reformatters","last_synced_at":"2026-04-11T04:03:32.044Z","repository":{"id":332212492,"uuid":"859043226","full_name":"dynamical-org/reformatters","owner":"dynamical-org","description":"Reformat weather datasets into Zarr","archived":false,"fork":false,"pushed_at":"2026-03-05T01:34:21.000Z","size":76041,"stargazers_count":26,"open_issues_count":24,"forks_count":6,"subscribers_count":5,"default_branch":"main","last_synced_at":"2026-03-05T01:42:04.025Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://dynamical.org/catalog","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dynamical-org.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2024-09-18T01:28:17.000Z","updated_at":"2026-03-04T22:28:11.000Z","dependencies_parsed_at":null,"dependency_job_id":"e434f263-2bd9-44ee-9b65-fc76f13fbc16","html_url":"https://github.com/dynamical-org/reformatters","commit_stats":null,"previous_names":["dynamical-org/reformatters"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/dynamical-org/reformatters","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dynamical-org%2Freformatters","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dynamical-org%2Freformatters/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dynamical-org%2Freformatters/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dynamical-org%2Freformatters/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dynamical-org","download_url":"https://codeload.github.com/dynamical-org/reformatters/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dynamical-org%2Freformatters/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30118932,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-05T09:35:22.236Z","status":"ssl_error","status_checked_at":"2026-03-05T09:35:20.028Z","response_time":93,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"owner":{"login":"dynamical-org","name":"dynamical-org","uuid":"156974863","kind":"organization","description":null,"email":null,"website":null,"location":null,"twitter":null,"company":null,"icon_url":"https://avatars.githubusercontent.com/u/156974863?v=4","repositories_count":1,"last_synced_at":"2024-01-20T21:33:46.041Z","metadata":{"has_sponsors_listing":false},"html_url":"https://github.com/dynamical-org","funding_links":[],"total_stars":null,"followers":null,"following":null,"created_at":"2024-01-20T21:33:46.044Z","updated_at":"2024-01-20T21:33:46.044Z","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dynamical-org","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dynamical-org/repositories"},"packages":[],"commits":{"id":11311122,"full_name":"dynamical-org/reformatters","default_branch":"master","total_commits":606,"total_committers":11,"total_bot_commits":4,"total_bot_committers":1,"mean_commits":55.09090909090909,"dds":0.3745874587458746,"past_year_total_commits":419,"past_year_total_committers":10,"past_year_total_bot_commits":4,"past_year_total_bot_committers":1,"past_year_mean_commits":41.9,"past_year_dds":0.3532219570405728,"last_synced_at":"2026-03-01T15:33:27.719Z","last_synced_commit":"e170d68f31fc528c2ca559132c10a0914da8d0e2","created_at":"2025-09-23T00:26:10.247Z","updated_at":"2026-03-01T15:33:14.275Z","committers":[{"name":"Alden Keefe Sampson","email":"aldenkeefesampson@gmail.com","login":"aldenks","count":379},{"name":"Alex","email":"mosegontar","login":"mosegontar","count":71},{"name":"Sam Neubardt","email":"sam@upstream.tech","login":"samn","count":56},{"name":"Margo Crawford","email":"margo@upstream.tech","login":"margocrawf","count":42},{"name":"Jack Kelly","email":"jack@OpenClimateFix.org","login":"JackKelly","count":20},{"name":"Dan Katz","email":"danielkatz125@gmail.com","login":"dan-katz","count":11},{"name":"Tony Cannistra","email":"tony.cannistra@gmail.com","login":"acannistra","count":8},{"name":"Lauren Gulland","email":"lauren@upstream.tech","login":"laurengulland","count":7},{"name":"Copilot","email":"198982749+Copilot","login":"Copilot","count":7},{"name":"dependabot[bot]","email":"49699333+dependabot[bot]","login":"dependabot[bot]","count":4},{"name":"Marshall","email":"mmoutenot@gmail.com","login":"mrshll","count":1}],"past_year_committers":[{"name":"Alden Keefe Sampson","email":"alden@dynamical.org","login":"aldenks","count":271},{"name":"Alex","email":"mosegontar","login":"mosegontar","count":71},{"name":"Sam Neubardt","email":"sam@upstream.tech","login":"samn","count":20},{"name":"Jack Kelly","email":"jack@OpenClimateFix.org","login":"JackKelly","count":20},{"name":"Dan Katz","email":"danielkatz125@gmail.com","login":"dan-katz","count":11},{"name":"Tony Cannistra","email":"tony.cannistra@gmail.com","login":"acannistra","count":8},{"name":"Copilot","email":"198982749+Copilot","login":"Copilot","count":7},{"name":"Lauren Gulland","email":"lauren@upstream.tech","login":"laurengulland","count":6},{"name":"dependabot[bot]","email":"49699333+dependabot[bot]","login":"dependabot[bot]","count":4},{"name":"Marshall","email":"mmoutenot@gmail.com","login":"mrshll","count":1}],"commits_url":"https://commits.ecosyste.ms/api/v1/hosts/GitHub/repositories/dynamical-org%2Freformatters/commits","host":{"name":"GitHub","url":"https://github.com","kind":"github","last_synced_at":"2026-03-03T00:02:26.129Z","repositories_count":6184080,"commits_count":930428817,"contributors_count":36037995,"owners_count":1146211,"icon_url":"https://github.com/github.png","host_url":"https://commits.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://commits.ecosyste.ms/api/v1/hosts/GitHub/repositories"}},"issues_stats":{"full_name":"dynamical-org/reformatters","html_url":"https://github.com/dynamical-org/reformatters","last_synced_at":"2026-03-03T03:08:03.598Z","status":null,"issues_count":8,"pull_requests_count":223,"avg_time_to_close_issue":null,"avg_time_to_close_pull_request":129073.0,"issues_closed_count":0,"pull_requests_closed_count":160,"pull_request_authors_count":8,"issue_authors_count":4,"avg_comments_per_issue":1.875,"avg_comments_per_pull_request":0.27802690582959644,"merged_pull_requests_count":155,"bot_issues_count":0,"bot_pull_requests_count":0,"past_year_issues_count":7,"past_year_pull_requests_count":165,"past_year_avg_time_to_close_issue":null,"past_year_avg_time_to_close_pull_request":121494.31481481482,"past_year_issues_closed_count":0,"past_year_pull_requests_closed_count":108,"past_year_pull_request_authors_count":6,"past_year_issue_authors_count":3,"past_year_avg_comments_per_issue":2.142857142857143,"past_year_avg_comments_per_pull_request":0.22424242424242424,"past_year_bot_issues_count":0,"past_year_bot_pull_requests_count":0,"past_year_merged_pull_requests_count":105,"created_at":"2025-07-16T12:49:38.101Z","updated_at":"2026-03-03T03:08:03.599Z","repository_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/repositories/dynamical-org%2Freformatters","issues_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/repositories/dynamical-org%2Freformatters/issues","issue_labels_count":{"enhancement":1},"pull_request_labels_count":{},"issue_author_associations_count":{"MEMBER":4,"NONE":2,"CONTRIBUTOR":2},"pull_request_author_associations_count":{"MEMBER":119,"CONTRIBUTOR":98,"NONE":6},"issue_authors":{"aldenks":3,"JackKelly":3,"briannapagan":1,"samn":1},"pull_request_authors":{"aldenks":113,"mosegontar":43,"samn":36,"dan-katz":10,"margocrawf":9,"acannistra":7,"JackKelly":3,"mrshll":2},"host":{"name":"GitHub","url":"https://github.com","kind":"github","last_synced_at":"2026-03-03T00:00:16.513Z","repositories_count":13554972,"issues_count":35039397,"pull_requests_count":114239445,"authors_count":11179430,"icon_url":"https://github.com/github.png","host_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/repositories","owners_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/owners","authors_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/authors"},"past_year_issue_labels_count":{"enhancement":1},"past_year_pull_request_labels_count":{},"past_year_issue_author_associations_count":{"MEMBER":4,"CONTRIBUTOR":2,"NONE":1},"past_year_pull_request_author_associations_count":{"MEMBER":97,"CONTRIBUTOR":64,"NONE":4},"past_year_issue_authors":{"aldenks":3,"JackKelly":3,"samn":1},"past_year_pull_request_authors":{"aldenks":91,"mosegontar":43,"samn":11,"dan-katz":10,"acannistra":7,"JackKelly":3},"maintainers":[{"login":"aldenks","count":116,"url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/authors/aldenks"},{"login":"samn","count":7,"url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/authors/samn"}],"active_maintainers":[{"login":"aldenks","count":94,"url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/authors/aldenks"},{"login":"samn","count":7,"url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/authors/samn"}]},"events":null,"keywords":[],"dependencies":[],"score":6.309918278226517,"created_at":"2025-09-23T00:25:39.587Z","updated_at":"2026-04-11T04:03:32.053Z","avatar_url":"https://github.com/dynamical-org.png","language":"Python","category":"Atmosphere","sub_category":"Meteorological Observation and Forecast","monthly_downloads":0,"total_dependent_repos":0,"total_dependent_packages":0,"readme":"# dynamical.org reformatters\nReformat weather datasets into zarr.\n\nBrowse the datasets produced by this repo at https://dynamical.org/catalog/.\n\n* See [AGENTS.md](AGENTS.md) for an overview of the approach and this repository.\n* [Integrate a new dataset](docs/dataset_integration_guide.md) to be reformatted.\n* [Add a new variable](docs/add_new_variable.md) to an existing dataset.\n\n## Local development\n\nWe use\n* `uv` to manage dependencies and python environments\n* `ruff` for linting and formatting\n* `ty` for type checking\n* `pytest` for testing\n* `prek` to automatically lint and format as you git commit\n\n### Setup\n1. [Install uv](https://docs.astral.sh/uv/getting-started/installation/)\n1. Run `uv run prek install` to setup the git hooks\n1. If you use VSCode, you may want to install the extensions (ruff) it will recommend when you open this folder\n\n### Running locally\n\n* `uv run main --help` - list all datasets\n* `uv run main \u003cDATASET_ID\u003e update-template`\n* `uv run main \u003cDATASET_ID\u003e backfill-local \u003cINIT_TIME_END\u003e`\n\n### Development commands\n* Add dependency: `uv add \u003cpackage\u003e [--dev]`. Use `--dev` to add a development only dependency.\n* Lint: `uv run ruff check [--fix]`\n* Type check: `uv run ty check`\n* Format: `uv run ruff format`\n* Tests: \n   * Run tests in parallel on all available cores: `uv run pytest`\n   * Run tests serially: `uv run pytest -n 0`\n\n## Deploying to the cloud\n\nTo reformat a large archive we parallelize work across multiple cloud servers.\n\nWe use\n* `docker` to package the code and dependencies\n* `kubernetes` indexed jobs to run work in parallel\n\n### Setup\n\n1. Install `docker` and `kubectl`. Make sure `docker` can be found at `/usr/bin/docker` and `kubectl` at `/usr/bin/kubectl`.\n1. Setup a docker image repository and export the `DOCKER_REPOSITORY` environment variable in your local shell. e.g. `export DOCKER_REPOSITORY=container.registry/\u003cproject-id\u003e/reformatters/main`. Follow your registry's instructions to allow your docker to authenticate and push images to the registry.\n1. Setup a kubernetes cluster and configure kubectl to point to your cluster. e.g. `aws eks update-kubeconfig --region \u003cregion\u003e --name \u003ccluster-name\u003e`, `gcloud container clusters get-credentials \u003ccluster-name\u003e --region \u003cregion\u003e --project \u003cproject\u003e`, etc.\n1. Create a kubectl secret containing a single json encoded value to be passed to fsspec `storage_options` or splatted as keyword arguments to an icechunk storage opener `kubectl create secret generic your-destination-storage-options-key --from-literal=contents='{\"key\": \"...\", \"secret\": \"...\"}'`. See `storage.py`.","funding_links":[],"readme_doi_urls":[],"works":{},"citation_counts":{},"total_citations":0,"keywords_from_contributors":[],"project_url":"https://ost.ecosyste.ms/api/v1/projects/325194","html_url":"https://ost.ecosyste.ms/projects/325194"}