Data

Tools

Indexes

Applications

Experiments

Open Sustainable Technology

reformatters

Reformat weather datasets into Zarr.
https://github.com/dynamical-org/reformatters

Category: Atmosphere
Sub Category: Meteorological Observation and Forecast

Last synced: about 23 hours ago
JSON representation

Repository metadata

Reformat weather datasets into Zarr

README.md

dynamical.org reformatters

Reformat weather datasets into zarr.

Browse the datasets produced by this repo at https://dynamical.org/catalog/.

Local development

We use

  • uv to manage dependencies and python environments
  • ruff for linting and formatting
  • ty for type checking
  • pytest for testing
  • prek to automatically lint and format as you git commit

Setup

  1. Install uv
  2. Run uv run prek install to setup the git hooks
  3. If you use VSCode, you may want to install the extensions (ruff) it will recommend when you open this folder

Running locally

  • uv run main --help - list all datasets
  • uv run main <DATASET_ID> update-template
  • uv run main <DATASET_ID> backfill-local <INIT_TIME_END>

Development commands

  • Add dependency: uv add <package> [--dev]. Use --dev to add a development only dependency.
  • Lint: uv run ruff check [--fix]
  • Type check: uv run ty check
  • Format: uv run ruff format
  • Tests:
    • Run tests in parallel on all available cores: uv run pytest
    • Run tests serially: uv run pytest -n 0

Deploying to the cloud

To reformat a large archive we parallelize work across multiple cloud servers.

We use

  • docker to package the code and dependencies
  • kubernetes indexed jobs to run work in parallel

Setup

  1. Install docker and kubectl. Make sure docker can be found at /usr/bin/docker and kubectl at /usr/bin/kubectl.
  2. Setup a docker image repository and export the DOCKER_REPOSITORY environment variable in your local shell. e.g. export DOCKER_REPOSITORY=container.registry/<project-id>/reformatters/main. Follow your registry's instructions to allow your docker to authenticate and push images to the registry.
  3. Setup a kubernetes cluster and configure kubectl to point to your cluster. e.g. aws eks update-kubeconfig --region <region> --name <cluster-name>, gcloud container clusters get-credentials <cluster-name> --region <region> --project <project>, etc.
  4. Create a kubectl secret containing a single json encoded value to be passed to fsspec storage_options or splatted as keyword arguments to an icechunk storage opener kubectl create secret generic your-destination-storage-options-key --from-literal=contents='{"key": "...", "secret": "..."}'. See storage.py.

Owner metadata

Committers metadata

Last synced: 19 days ago

Total Commits: 606
Total Committers: 11
Avg Commits per committer: 55.091
Development Distribution Score (DDS): 0.375

Commits in past year: 419
Committers in past year: 10
Avg Commits per committer in past year: 41.9
Development Distribution Score (DDS) in past year: 0.353

Name Email Commits
Alden Keefe Sampson a****n@g****m 379
Alex m****r 71
Sam Neubardt s****m@u****h 56
Margo Crawford m****o@u****h 42
Jack Kelly j****k@O****g 20
Dan Katz d****5@g****m 11
Tony Cannistra t****a@g****m 8
Lauren Gulland l****n@u****h 7
Copilot 1****t 7
dependabot[bot] 4****] 4
Marshall m****t@g****m 1

Committer domains:

Issue and Pull Request metadata

Last synced: 18 days ago

Total issues: 8
Total pull requests: 223
Average time to close issues: N/A
Average time to close pull requests: 1 day
Total issue authors: 4
Total pull request authors: 8
Average comments per issue: 1.88
Average comments per pull request: 0.28
Merged pull request: 155
Bot issues: 0
Bot pull requests: 0

Past year issues: 7
Past year pull requests: 165
Past year average time to close issues: N/A
Past year average time to close pull requests: 1 day
Past year issue authors: 3
Past year pull request authors: 6
Past year average comments per issue: 2.14
Past year average comments per pull request: 0.22
Past year merged pull request: 105
Past year bot issues: 0
Past year bot pull requests: 0

More stats: https://issues.ecosyste.ms/repositories/lookup?url=https://github.com/dynamical-org/reformatters

Top Issue Authors

  • aldenks (3)
  • JackKelly (3)
  • briannapagan (1)
  • samn (1)

Top Pull Request Authors

  • aldenks (113)
  • mosegontar (43)
  • samn (36)
  • dan-katz (10)
  • margocrawf (9)
  • acannistra (7)
  • JackKelly (3)
  • mrshll (2)

Top Issue Labels

  • enhancement (1)

Top Pull Request Labels

Score: 6.309918278226517