A curated list of open technology projects to sustain a stable climate, energy supply, biodiversity and natural resources.

Herbie

A python package that downloads recent and archived numerical weather prediction model output from different cloud archive sources.
https://github.com/blaylockbk/Herbie

Category: Atmosphere
Sub Category: Meteorological Observation and Forecast

Keywords

big-data-program cfgrib download ecmwf-data gfs grib grib2 hrrr noaa-data nomads numerical-weather-prediction open-data python rap xarray

Keywords from Contributors

weather charts simulations share observational polars synoptic-api mesowest mesonet streams

Last synced: about 13 hours ago
JSON representation

Repository metadata

Download numerical weather prediction datasets (HRRR, RAP, GFS, IFS, etc.) from NOMADS, NODD partners (Amazon, Google, Microsoft), ECMWF open data, and the University of Utah Pando Archive System.

README.md

Herbie: Retrieve NWP Model Data ๐Ÿ

PyPI - Python Version
PyPI
Conda Version

DOI

Tests (Conda)
Tests (Python)
Documentation Status
Conda Recipe
Ruff

Conda Downloads
PyPI - Downloads
PyPI - License

๐Ÿ“˜ Documentation | ๐Ÿ’ฌ Discussions | โ” Ask For Help

Herbie is a python package that downloads recent and archived numerical weather prediction (NWP) model output from different cloud archive sources. NWP data is distributed in GRIB2 format which Herbie reads using xarray+cfgrib. Herbie also provides some extra features to help visualize and extract data.

Herbie helps you discover, download, and read data from:

Much of this data is made available through the NOAA Open Data Dissemination (NODD) program which has made weather data more accessible than ever before.

Installation

conda or mamba

The easiest way to instal Herbie and its dependencies is with Conda from conda-forge.

conda install -c conda-forge herbie-data

You may also create the provided Conda environment, environment.yml.

# Download environment file
wget https://github.com/blaylockbk/Herbie/raw/main/environment.yml

# Modify that file if you wish.

# Create the environment
conda env create -f environment.yml

# Activate the environment
conda activate herbie

pip

Alternatively, Herbie is published on PyPI and you can install it with pip.

# Latest published version
pip install herbie-data

# ~~ or ~~

# Most recent changes
pip install git+https://github.com/blaylockbk/Herbie.git

# Dependecies for extra features
pip install herbie-data[extra]

Keep in mind that Herbie does require the following dependencies that you will have to install yourself if you don't have them in your PATH:

  • Python 3.10+
  • cURL
  • wgrib2 (for some extra features)

uv

You can add Herbie to your project with the uv command:

uv add herbie-data

Capabilities

  • Search for model output from different data sources.
  • Download full GRIB2 files.
  • Download subset GRIB2 files (by grib field).
  • Read data with xarray.
  • Read index file with Pandas.
  • Extra features (herbie xarray accessors)
    • Extract data at a point
    • Get Cartopy coordinate references system
    • Plot data with Cartopy (very early development).
  graph TD;
      d1[(HRRR)] -..-> H
      d2[(RAP)] -.-> H
      d3[(GFS)] -..-> H
      d33[(GEFS)] -.-> H
      d4[(IFS)] -..-> H
      d44[(AIFS)] -..-> H
      d5[(NBM)] -.-> H
      d6[(RRFS)] -..-> H
      d7[(RTMA)] -.-> H
      d8[(URMA)] -..-> H
      H((Herbie))
      H --- .inventory
      H --- .download
      H --- .xarray

      style H fill:#d8c89d,stroke:#0c3576,stroke-width:4px,color:#000000
from herbie import Herbie

# Herbie object for the HRRR model 6-hr surface forecast product
H = Herbie(
  '2021-01-01 12:00',
  model='hrrr',
  product='sfc',
  fxx=6
)

# Look at file contents
H.inventory()

# Download the full GRIB2 file
H.download()

# Download a subset, like all fields at 500 mb
H.download(":500 mb")

# Read subset with xarray, like 2-m temperature.
H.xarray("TMP:2 m")

Data Sources

Herbie downloads model data from the following sources, but can be extended to include others:

Community

Having trouble using Herbie or have a question? โ” GitHub Discussions/Ask For Help

Just want to talk about Herbie or have an idea? ๐Ÿ’ฌ GitHub Discussions

See something that might be wrong? ๐Ÿš‘ GitHub Issues

Want to contribute? Great! I'd love your help.

How to Cite and Acknowledge

If Herbie played an important role in your work, please tell me about it! Also, consider including a citation or acknowledgement in your article or product.

Suggested Citation

Blaylock, B. K. (YEAR). Herbie: Retrieve Numerical Weather Prediction Model Data (Version 20xx.x.x) [Computer software]. https://doi.org/10.5281/zenodo.4567540

Suggested Acknowledgment

A portion of this work used code generously provided by Brian Blaylock's Herbie python package (Version 20xx.x.x) (https://doi.org/10.5281/zenodo.4567540)

History

During my PhD at the University of Utah, I created, at the time, the only publicly-accessible archive of HRRR data. Over 1,000 research scientists and professionals used that archive.

Herbie was then developed to access HRRR data from that archive and was first used on the Open Science Grid.

In the later half of 2020, the HRRR dataset from 2014 to present was made available through the NODD Open Data Dissemination Program (formerly NOAA's Big Data Program). The latest version of Herbie organizes and expands my original download scripts into a more coherent package with the extended ability to download data for other models from many different archive sources, and it will continues to evolve.

I originally released this package under the name โ€œHRRR-Bโ€ because it only worked with the HRRR dataset; the โ€œBโ€ was for Brian. Since then, I have added the ability to download many more models including RAP, GFS, ECMWF, GEFS, and RRFS with the potential to add more models in the future. Thus, this package was renamed Herbie, named after one of my favorite childhood movies.

The University of Utah MesoWest group now manages a HRRR archive in Zarr format. Maybe someday, Herbie will be able to take advantage of that archive.

Thanks for using Herbie, and happy racing!

๐Ÿ Brian

P.S. If you like Herbie, check out my other repos:

  • ๐ŸŒŽ GOES-2-go: A python package to download GOES-East/West data and make RGB composites.
  • ๐ŸŒก SynopticPy: A python package to download mesonet data from the Synoptic API.
  • ๐Ÿ”จ Carpenter Workshop: A python package with various tools I made that are useful (like easy funxtions to build Cartopy maps).
  • ๐Ÿ’ฌ Bubble Print: A silly little python package that gives your print statement's personality.
  • ๐Ÿ“œ MET Syntax: An extension for Visual Studio Code that gives syntax highlighting for Model Evaluation Tools (MET) configuration files.

Note: Alternative Download Tools
As an alternative to Herbie, you can use rclone to download files from AWS or GCP. I love rclone. Here is a short rclone tutorial

| Visualize Structure | Star History | PyPI Download Statistics

Citation (CITATION.cff)

cff-version: 1.2.0
abstract: Herbie is a python package that downloads recent and archived numerical weather prediction (NWP) model output from different cloud archive sources. Its most popular capability is to download HRRR model data. NWP data in GRIB2 format can be read into an xarray dataframe using the cfgrib package. Much of this data is made available through the NOAA Open Data Dissemination (NODD) Program (formerly the Big Data Program) which has made weather data more accessible than ever before.
message: "If you use this software, please cite it as below."
authors:
  - family-names: Blaylock
    given-names: Brian K.
    email: [email protected]
    orcid: "https://orcid.org/0000-0003-2133-9313"
title: "Herbie: Retrieve Numerical Weather Prediction Model Data"
version: 2024.5.0
date-released: "2023-03-02"
url: "https://herbie.readthedocs.io/"
repository-code: "https://github.com/blaylockbk/Herbie"
type: software
keywords:
  - meteorology
  - weather
  - numerical weather prediction
  - forecast
  - atmosphere
license: "MIT"
identifiers:
  - type: doi
    value: 10.5281/zenodo.4567540

Owner metadata


GitHub Events

Total
Last Year

Committers metadata

Last synced: 4 days ago

Total Commits: 1,202
Total Committers: 25
Avg Commits per committer: 48.08
Development Distribution Score (DDS): 0.101

Commits in past year: 184
Committers in past year: 8
Avg Commits per committer in past year: 23.0
Development Distribution Score (DDS) in past year: 0.19

Name Email Commits
Brian Blaylock b****k@g****m 1081
Coat a****t@a****r 33
Andreas Motl a****l@p****g 18
Brian Blaylock b****r@n****l 13
karlwx k****2@p****u 13
Rafael Guedes r****s@o****e 12
GabrielKS 2****S 4
Haim Daniel h****l@g****m 4
Tamas Weisz w****a@g****m 3
Will Hobbs 4****s 3
Haim Daniel h****m@j****m 2
[email protected] d****e@g****m 2
fleegs79 1****9 2
Alexander Rey m****l@a****a 1
Hao Lyu 2****u 1
Steve Nesbitt s****t@i****u 1
Taylor Mandelbaum m****r@g****m 1
The Gitter Badger b****r@g****m 1
Xuesong Wang x****g@w****u 1
cyril 3****s 1
joshuaeh j****2@g****m 1
timothydonohue t****m@a****i 1
David Landry d****y@i****r 1
Brian = 1
Stephen Campbell s****0@h****m 1

Committer domains:


Issue and Pull Request metadata

Last synced: 1 day ago

Total issues: 208
Total pull requests: 109
Average time to close issues: 2 months
Average time to close pull requests: 16 days
Total issue authors: 70
Total pull request authors: 21
Average comments per issue: 2.03
Average comments per pull request: 0.96
Merged pull request: 92
Bot issues: 0
Bot pull requests: 0

Past year issues: 70
Past year pull requests: 39
Past year average time to close issues: 12 days
Past year average time to close pull requests: 7 days
Past year issue authors: 30
Past year pull request authors: 9
Past year average comments per issue: 1.77
Past year average comments per pull request: 0.9
Past year merged pull request: 30
Past year bot issues: 0
Past year bot pull requests: 0

More stats: https://issues.ecosyste.ms/repositories/lookup?url=https://github.com/blaylockbk/Herbie

Top Issue Authors

  • blaylockbk (95)
  • williamhobbs (13)
  • jp2nyy (6)
  • btickell (4)
  • amotl (4)
  • karlwx (4)
  • byphilipp (3)
  • rafa-guedes (3)
  • peterdudfield (3)
  • SaundersJE97 (2)
  • fleegs79 (2)
  • cole-p (2)
  • svarlamov (2)
  • RBhupi (2)
  • bryanguarente (2)

Top Pull Request Authors

  • blaylockbk (70)
  • amotl (6)
  • karlwx (5)
  • williamhobbs (5)
  • alcoat (4)
  • haim0n (3)
  • rafa-guedes (2)
  • davidlandry93 (1)
  • joshuaeh (1)
  • Davidxswang (1)
  • sjcrz (1)
  • cdibble (1)
  • timdonohue-aerology (1)
  • aaTman (1)
  • fleegs79 (1)

Top Issue Labels

  • help wanted (21)
  • enhancement (16)
  • documentation (15)
  • ECMWF (11)
  • ๐Ÿ’ก Idea (8)
  • bug (7)
  • GFS (3)
  • RAP (3)
  • CFS (2)
  • upstream (2)
  • question (2)
  • ๐Ÿ Canada (2)
  • good first issue (2)
  • accessors (2)
  • NAM (1)
  • FastHerbie (1)
  • ๐Ÿงช CI/CD (1)
  • inventory (1)
  • RRFS (1)
  • GEFS (1)
  • NBM (1)
  • wontfix (1)
  • needs more info (1)
  • ๐Ÿ”Ž searchString (1)
  • windows (1)

Top Pull Request Labels

  • enhancement (4)
  • ECMWF (3)
  • help wanted (1)
  • HAFS (1)
  • install (1)
  • RRFS (1)
  • GEFS (1)

Package metadata

pypi.org: herbie-data

Download numerical weather prediction GRIB2 model data.

  • Homepage: https://github.com/blaylockbk/Herbie
  • Documentation: https://herbie.readthedocs.io/
  • Licenses: MIT License Copyright (c) 2019-2024 Brian K. Blaylock Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
  • Latest release: 2025.3.1 (published about 1 month ago)
  • Last Synced: 2025-04-25T13:05:52.749Z (1 day ago)
  • Versions: 21
  • Dependent Packages: 3
  • Dependent Repositories: 1
  • Downloads: 56,468 Last month
  • Rankings:
    • Stargazers count: 3.683%
    • Forks count: 5.604%
    • Downloads: 6.649%
    • Dependent packages count: 7.31%
    • Average: 9.067%
    • Dependent repos count: 22.088%
  • Maintainers (1)
pypi.org: hrrrb

Download model data (HRRR, RAP, GFS, NBM, etc.) from NOMADS, NOAA's Big Data Program partners (Amazon, Google, Microsoft), and the University of Utah Pando Archive System.

  • Homepage:
  • Documentation: https://blaylockbk.github.io/Herbie/_build/html/
  • Licenses: MIT
  • Latest release: 0.0.6 (published over 3 years ago)
  • Last Synced: 2025-04-25T13:05:52.490Z (1 day ago)
  • Versions: 6
  • Dependent Packages: 0
  • Dependent Repositories: 1
  • Downloads: 498 Last month
  • Rankings:
    • Stargazers count: 3.683%
    • Forks count: 5.604%
    • Dependent packages count: 7.31%
    • Average: 17.182%
    • Dependent repos count: 22.088%
    • Downloads: 47.224%
  • Maintainers (1)
conda-forge.org: herbie-data

Herbie is a python package that downloads recent and archived numerical weather prediction (NWP) model output from different cloud archive sources. Its most popular capability is to download HRRR model data. NWP data in GRIB2 format can be read with xarray+cfgrib. Much of this data is made available through the NOAA Open Data Dissemination (NODD) Program (formerly the Big Data Program) which has made weather data more accessible than ever before.

  • Homepage: https://github.com/blaylockbk/Herbie
  • Licenses: MIT
  • Latest release: 2022.9.0.post1 (published over 2 years ago)
  • Last Synced: 2025-04-01T02:10:15.489Z (26 days ago)
  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 1
  • Rankings:
    • Dependent repos count: 24.103%
    • Stargazers count: 24.852%
    • Forks count: 26.733%
    • Average: 31.807%
    • Dependent packages count: 51.54%

Dependencies

ci/requirements.txt pypi
  • cfgrib >=0.9.9.1
  • metpy >=1.3.0
  • numpy >=1.22.3
  • pandas >=1.4.1
  • requests >=2.27.1
  • toml >=0.10.2
  • xarray >=2022.3.0
requirements.txt pypi
  • cartopy *
  • cfgrib *
  • matplotlib *
  • metpy *
  • numpy *
  • pandas *
  • pygrib *
  • toml *
  • xarray *
setup.py pypi
  • cartopy *
  • cfgrib *
  • metpy *
  • numpy *
  • pandas *
  • pygrib *
  • toml *
  • xarray *
.github/workflows/black.yml actions
  • actions/checkout v2 composite
  • psf/black stable composite
.github/workflows/python-package-conda.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v3 composite
.github/workflows/tests-conda.yml actions
  • actions/cache v3 composite
  • actions/checkout v3 composite
  • conda-incubator/setup-miniconda v2 composite
.github/workflows/tests-python.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v4 composite
requirements-test.txt pypi
  • pytest <8 test
ci/environment.yml pypi
environment.yml pypi
pyproject.toml pypi

Score: 20.656892593735165