Data

Tools

Indexes

Applications

Experiments

Open Sustainable Technology

reformatters

Reformat weather datasets into Zarr.
https://github.com/dynamical-org/reformatters

Category: Atmosphere
Sub Category: Meteorological Observation and Forecast

Last synced: about 6 hours ago
JSON representation

Repository metadata

Reformat weather datasets into Zarr

README.md

dynamical.org reformatters

Reformat weather datasets into zarr.

Browse the datasets produced by this repo at https://dynamical.org/catalog/.

DOI

Local development

We use

  • uv to manage dependencies and python environments
  • ruff for linting and formatting
  • ty for type checking
  • pytest for testing
  • prek to automatically lint and format as you git commit

Setup

  1. Install uv
  2. Run uv run prek install to setup the git hooks
  3. If you use VSCode, you may want to install the extensions (ruff) it will recommend when you open this folder

Running locally

  • uv run main --help - list all datasets
  • uv run main <DATASET_ID> update-template
  • uv run main <DATASET_ID> backfill-local <APPEND_DIM_END>

Development commands

  • Add dependency: uv add <package> [--dev]. Use --dev to add a development only dependency.
  • Lint: uv run ruff check [--fix]
  • Type check: uv run ty check
  • Format: uv run ruff format
  • Tests:
    • Run tests in parallel on all available cores: uv run pytest
    • Run tests serially: uv run pytest -n 0

Deploying to the cloud

To reformat a large archive we parallelize work across multiple cloud servers.

We use

  • docker to package the code and dependencies
  • kubernetes indexed jobs to run work in parallel

Setup

  1. Install docker and kubectl. Make sure docker can be found at /usr/bin/docker and kubectl at /usr/bin/kubectl.
  2. Setup a docker image repository and export the DOCKER_REPOSITORY environment variable in your local shell. e.g. export DOCKER_REPOSITORY=container.registry/<project-id>/reformatters/main. Follow your registry's instructions to allow your docker to authenticate and push images to the registry.
  3. Setup a kubernetes cluster and configure kubectl to point to your cluster. e.g. aws eks update-kubeconfig --region <region> --name <cluster-name>, gcloud container clusters get-credentials <cluster-name> --region <region> --project <project>, etc.
  4. Create a kubectl secret containing a single json encoded value to be passed to fsspec storage_options or splatted as keyword arguments to an icechunk storage opener kubectl create secret generic your-destination-storage-options-key --from-literal=contents='{"key": "...", "secret": "..."}'. See storage.py.

Citation (CITATION.cff)

cff-version: 1.2.0
message: "If you use dynamical.org data, please cite as below."
title: "dynamical.org reformatters"
type: software
doi: 10.5281/zenodo.18777399
url: "https://dynamical.org"
repository-code: "https://github.com/dynamical-org/reformatters"
license: BSD-3-Clause
authors:
  - family-names: Keefe Sampson
    given-names: Alden
  - family-names: Moutenot
    given-names: Marshall
  - name: "dynamical.org contributors"

Owner metadata

GitHub Events

Total
Last Year

Committers metadata

Last synced: 1 day ago

Total Commits: 877
Total Committers: 11
Avg Commits per committer: 79.727
Development Distribution Score (DDS): 0.316

Commits in past year: 500
Committers in past year: 9
Avg Commits per committer in past year: 55.556
Development Distribution Score (DDS) in past year: 0.27

Name Email Commits
Alden Keefe Sampson a****n@d****g 600
Alex m****r 71
Sam Neubardt s****m@u****h 56
Margo Crawford m****o@u****h 42
Marshall m****l@d****g 37
Jack Kelly j****k@O****g 20
dependabot[bot] 4****] 17
Dan Katz d****5@g****m 11
Tony Cannistra t****a@g****m 8
Copilot 1****t 8
Lauren Gulland l****n@u****h 7

Committer domains:

Issue and Pull Request metadata

Last synced: 1 day ago

Total issues: 50
Total pull requests: 324
Average time to close issues: about 1 month
Average time to close pull requests: 2 days
Total issue authors: 5
Total pull request authors: 10
Average comments per issue: 3.06
Average comments per pull request: 0.31
Merged pull request: 230
Bot issues: 0
Bot pull requests: 16

Past year issues: 46
Past year pull requests: 159
Past year average time to close issues: 19 days
Past year average time to close pull requests: 2 days
Past year issue authors: 4
Past year pull request authors: 7
Past year average comments per issue: 2.59
Past year average comments per pull request: 0.35
Past year merged pull request: 108
Past year bot issues: 0
Past year bot pull requests: 16

More stats: https://issues.ecosyste.ms/repositories/lookup?url=https://github.com/dynamical-org/reformatters

Top Issue Authors

  • mrshll (29)
  • aldenks (14)
  • JackKelly (5)
  • briannapagan (1)
  • samn (1)

Top Pull Request Authors

  • aldenks (150)
  • mrshll (49)
  • mosegontar (43)
  • samn (36)
  • dependabot[bot] (16)
  • dan-katz (10)
  • margocrawf (9)
  • acannistra (7)
  • JackKelly (3)
  • mayadebellis (1)

Top Issue Labels

  • enhancement (14)
  • bug (1)

Top Pull Request Labels

  • dependencies (16)
  • python:uv (14)
  • github_actions (2)
  • codex (1)

Score: 6.715383386334681