Herbie
A python package that downloads recent and archived numerical weather prediction model output from different cloud archive sources.
https://github.com/blaylockbk/Herbie
Category: Atmosphere
Sub Category: Meteorological Observation and Forecast
Keywords
big-data-program cfgrib download ecmwf-data gfs grib grib2 hrrr noaa-data nomads numerical-weather-prediction open-data python rap xarray
Keywords from Contributors
weather charts simulations share observational polars synoptic-api mesowest mesonet streams
Last synced: about 13 hours ago
JSON representation
Repository metadata
Download numerical weather prediction datasets (HRRR, RAP, GFS, IFS, etc.) from NOMADS, NODD partners (Amazon, Google, Microsoft), ECMWF open data, and the University of Utah Pando Archive System.
- Host: GitHub
- URL: https://github.com/blaylockbk/Herbie
- Owner: blaylockbk
- License: mit
- Created: 2020-06-26T17:43:11.000Z (almost 5 years ago)
- Default Branch: main
- Last Pushed: 2025-04-17T04:17:49.000Z (10 days ago)
- Last Synced: 2025-04-20T10:41:45.216Z (7 days ago)
- Topics: big-data-program, cfgrib, download, ecmwf-data, gfs, grib, grib2, hrrr, noaa-data, nomads, numerical-weather-prediction, open-data, python, rap, xarray
- Language: Python
- Homepage: https://herbie.readthedocs.io/
- Size: 172 MB
- Stars: 579
- Watchers: 16
- Forks: 87
- Open Issues: 78
- Releases: 24
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Citation: CITATION.cff
README.md
Herbie: Retrieve NWP Model Data ๐
๐ Documentation | ๐ฌ Discussions | โ Ask For Help
Herbie is a python package that downloads recent and archived numerical weather prediction (NWP) model output from different cloud archive sources. NWP data is distributed in GRIB2 format which Herbie reads using xarray+cfgrib. Herbie also provides some extra features to help visualize and extract data.
Herbie helps you discover, download, and read data from:
- High Resolution Rapid Refresh (HRRR) | HRRR-Alaska
- Rapid Refresh (RAP)
- Global Forecast System (GFS)
- Global Ensemble Forecast System (GEFS)
- ECMWF Open Data Forecasts (IFS and AIFS)
- Navy Global Environmental Model (NAVGEM)
- North American Mesoscale Model (NAM)
- National Blend of Models (NBM)
- Rapid Refresh Forecast System (RRFS) prototype
- Real-Time/Un-Restricted Mesoscale Analysis (RTMA/URMA)
- Hurricane Analysis And Forecast System (HAFS)
- High Resolution Deterministic Prediction System (HRDPS)
- Climate Forecast System (CFS)
- and more! Check out the gallery.
Much of this data is made available through the NOAA Open Data Dissemination (NODD) program which has made weather data more accessible than ever before.
Installation
conda
or mamba
The easiest way to instal Herbie and its dependencies is with Conda from conda-forge.
conda install -c conda-forge herbie-data
You may also create the provided Conda environment, environment.yml
.
# Download environment file
wget https://github.com/blaylockbk/Herbie/raw/main/environment.yml
# Modify that file if you wish.
# Create the environment
conda env create -f environment.yml
# Activate the environment
conda activate herbie
pip
Alternatively, Herbie is published on PyPI and you can install it with pip.
# Latest published version
pip install herbie-data
# ~~ or ~~
# Most recent changes
pip install git+https://github.com/blaylockbk/Herbie.git
# Dependecies for extra features
pip install herbie-data[extra]
Keep in mind that Herbie does require the following dependencies that you will have to install yourself if you don't have them in your PATH:
uv
You can add Herbie to your project with the uv command:
uv add herbie-data
Capabilities
- Search for model output from different data sources.
- Download full GRIB2 files.
- Download subset GRIB2 files (by grib field).
- Read data with xarray.
- Read index file with Pandas.
- Extra features (herbie xarray accessors)
- Extract data at a point
- Get Cartopy coordinate references system
- Plot data with Cartopy (very early development).
graph TD;
d1[(HRRR)] -..-> H
d2[(RAP)] -.-> H
d3[(GFS)] -..-> H
d33[(GEFS)] -.-> H
d4[(IFS)] -..-> H
d44[(AIFS)] -..-> H
d5[(NBM)] -.-> H
d6[(RRFS)] -..-> H
d7[(RTMA)] -.-> H
d8[(URMA)] -..-> H
H((Herbie))
H --- .inventory
H --- .download
H --- .xarray
style H fill:#d8c89d,stroke:#0c3576,stroke-width:4px,color:#000000
from herbie import Herbie
# Herbie object for the HRRR model 6-hr surface forecast product
H = Herbie(
'2021-01-01 12:00',
model='hrrr',
product='sfc',
fxx=6
)
# Look at file contents
H.inventory()
# Download the full GRIB2 file
H.download()
# Download a subset, like all fields at 500 mb
H.download(":500 mb")
# Read subset with xarray, like 2-m temperature.
H.xarray("TMP:2 m")
Data Sources
Herbie downloads model data from the following sources, but can be extended to include others:
- NOMADS
- NOAA Open Data Dissemination Program (NODD) partners (i.e., AWS, Google, Azure).
- ECMWF Open Data Forecasts
- University of Utah CHPC Pando archive
- Local file system
Community
Having trouble using Herbie or have a question? โ GitHub Discussions/Ask For Help
Just want to talk about Herbie or have an idea? ๐ฌ GitHub Discussions
See something that might be wrong? ๐ GitHub Issues
Want to contribute? Great! I'd love your help.
- "Watch" this repo's discussions and issues.
- Participate in ๐ฌ GitHub Discussions and answer questions.
- Share how you use Herbie in ๐ GitHub Discussions/Show and tell
- Open an issue or file a pull request to make Herbie even better!
- Contribute to documentation.
- Test latest releases and report issues.
- Read the ๐จ๐ปโ๐ป Disclaimer & Contributing Guide
How to Cite and Acknowledge
If Herbie played an important role in your work, please tell me about it! Also, consider including a citation or acknowledgement in your article or product.
Suggested Citation
Blaylock, B. K. (YEAR). Herbie: Retrieve Numerical Weather Prediction Model Data (Version 20xx.x.x) [Computer software]. https://doi.org/10.5281/zenodo.4567540
Suggested Acknowledgment
A portion of this work used code generously provided by Brian Blaylock's Herbie python package (Version 20xx.x.x) (https://doi.org/10.5281/zenodo.4567540)
History
During my PhD at the University of Utah, I created, at the time, the only publicly-accessible archive of HRRR data. Over 1,000 research scientists and professionals used that archive.
Herbie was then developed to access HRRR data from that archive and was first used on the Open Science Grid.
In the later half of 2020, the HRRR dataset from 2014 to present was made available through the NODD Open Data Dissemination Program (formerly NOAA's Big Data Program). The latest version of Herbie organizes and expands my original download scripts into a more coherent package with the extended ability to download data for other models from many different archive sources, and it will continues to evolve.
I originally released this package under the name โHRRR-Bโ because it only worked with the HRRR dataset; the โBโ was for Brian. Since then, I have added the ability to download many more models including RAP, GFS, ECMWF, GEFS, and RRFS with the potential to add more models in the future. Thus, this package was renamed Herbie, named after one of my favorite childhood movies.
The University of Utah MesoWest group now manages a HRRR archive in Zarr format. Maybe someday, Herbie will be able to take advantage of that archive.
Thanks for using Herbie, and happy racing!
๐ Brian
P.S. If you like Herbie, check out my other repos:
- ๐ GOES-2-go: A python package to download GOES-East/West data and make RGB composites.
- ๐ก SynopticPy: A python package to download mesonet data from the Synoptic API.
- ๐จ Carpenter Workshop: A python package with various tools I made that are useful (like easy funxtions to build Cartopy maps).
- ๐ฌ Bubble Print: A silly little python package that gives your print statement's personality.
- ๐ MET Syntax: An extension for Visual Studio Code that gives syntax highlighting for Model Evaluation Tools (MET) configuration files.
Note: Alternative Download Tools
As an alternative to Herbie, you can use rclone to download files from AWS or GCP. I love rclone. Here is a short rclone tutorial
| Visualize Structure | Star History | PyPI Download Statistics
Citation (CITATION.cff)
cff-version: 1.2.0 abstract: Herbie is a python package that downloads recent and archived numerical weather prediction (NWP) model output from different cloud archive sources. Its most popular capability is to download HRRR model data. NWP data in GRIB2 format can be read into an xarray dataframe using the cfgrib package. Much of this data is made available through the NOAA Open Data Dissemination (NODD) Program (formerly the Big Data Program) which has made weather data more accessible than ever before. message: "If you use this software, please cite it as below." authors: - family-names: Blaylock given-names: Brian K. email: [email protected] orcid: "https://orcid.org/0000-0003-2133-9313" title: "Herbie: Retrieve Numerical Weather Prediction Model Data" version: 2024.5.0 date-released: "2023-03-02" url: "https://herbie.readthedocs.io/" repository-code: "https://github.com/blaylockbk/Herbie" type: software keywords: - meteorology - weather - numerical weather prediction - forecast - atmosphere license: "MIT" identifiers: - type: doi value: 10.5281/zenodo.4567540
Owner metadata
- Name: Brian Blaylock
- Login: blaylockbk
- Email:
- Kind: user
- Description: Meteorologist
- Website: http://home.chpc.utah.edu/~u0553130/Brian_Blaylock/home.html
- Location: Monterey, CA
- Twitter: blaylockbk
- Company:
- Icon url: https://avatars.githubusercontent.com/u/6249613?u=83e593317387e0cfacc6d86df40f6d2ee2f13676&v=4
- Repositories: 45
- Last ynced at: 2024-03-15T18:11:46.924Z
- Profile URL: https://github.com/blaylockbk
GitHub Events
Total
- Create event: 16
- Release event: 4
- Issues event: 35
- Watch event: 94
- Delete event: 4
- Issue comment event: 81
- Push event: 92
- Pull request review comment event: 3
- Pull request review event: 5
- Pull request event: 24
- Fork event: 15
Last Year
- Create event: 16
- Release event: 4
- Issues event: 35
- Watch event: 94
- Delete event: 4
- Issue comment event: 81
- Push event: 92
- Pull request review comment event: 3
- Pull request review event: 5
- Pull request event: 24
- Fork event: 15
Committers metadata
Last synced: 4 days ago
Total Commits: 1,202
Total Committers: 25
Avg Commits per committer: 48.08
Development Distribution Score (DDS): 0.101
Commits in past year: 184
Committers in past year: 8
Avg Commits per committer in past year: 23.0
Development Distribution Score (DDS) in past year: 0.19
Name | Commits | |
---|---|---|
Brian Blaylock | b****k@g****m | 1081 |
Coat | a****t@a****r | 33 |
Andreas Motl | a****l@p****g | 18 |
Brian Blaylock | b****r@n****l | 13 |
karlwx | k****2@p****u | 13 |
Rafael Guedes | r****s@o****e | 12 |
GabrielKS | 2****S | 4 |
Haim Daniel | h****l@g****m | 4 |
Tamas Weisz | w****a@g****m | 3 |
Will Hobbs | 4****s | 3 |
Haim Daniel | h****m@j****m | 2 |
[email protected] | d****e@g****m | 2 |
fleegs79 | 1****9 | 2 |
Alexander Rey | m****l@a****a | 1 |
Hao Lyu | 2****u | 1 |
Steve Nesbitt | s****t@i****u | 1 |
Taylor Mandelbaum | m****r@g****m | 1 |
The Gitter Badger | b****r@g****m | 1 |
Xuesong Wang | x****g@w****u | 1 |
cyril | 3****s | 1 |
joshuaeh | j****2@g****m | 1 |
timothydonohue | t****m@a****i | 1 |
David Landry | d****y@i****r | 1 |
Brian | = | 1 |
Stephen Campbell | s****0@h****m | 1 |
Committer domains:
- inria.fr: 1
- aerology.ai: 1
- wayne.edu: 1
- gitter.im: 1
- illinois.edu: 1
- alexanderrey.ca: 1
- jether-energy.com: 1
- oceanum.science: 1
- psu.edu: 1
- nrlmry.navy.mil: 1
- panodata.org: 1
- actimar.fr: 1
Issue and Pull Request metadata
Last synced: 1 day ago
Total issues: 208
Total pull requests: 109
Average time to close issues: 2 months
Average time to close pull requests: 16 days
Total issue authors: 70
Total pull request authors: 21
Average comments per issue: 2.03
Average comments per pull request: 0.96
Merged pull request: 92
Bot issues: 0
Bot pull requests: 0
Past year issues: 70
Past year pull requests: 39
Past year average time to close issues: 12 days
Past year average time to close pull requests: 7 days
Past year issue authors: 30
Past year pull request authors: 9
Past year average comments per issue: 1.77
Past year average comments per pull request: 0.9
Past year merged pull request: 30
Past year bot issues: 0
Past year bot pull requests: 0
Top Issue Authors
- blaylockbk (95)
- williamhobbs (13)
- jp2nyy (6)
- btickell (4)
- amotl (4)
- karlwx (4)
- byphilipp (3)
- rafa-guedes (3)
- peterdudfield (3)
- SaundersJE97 (2)
- fleegs79 (2)
- cole-p (2)
- svarlamov (2)
- RBhupi (2)
- bryanguarente (2)
Top Pull Request Authors
- blaylockbk (70)
- amotl (6)
- karlwx (5)
- williamhobbs (5)
- alcoat (4)
- haim0n (3)
- rafa-guedes (2)
- davidlandry93 (1)
- joshuaeh (1)
- Davidxswang (1)
- sjcrz (1)
- cdibble (1)
- timdonohue-aerology (1)
- aaTman (1)
- fleegs79 (1)
Top Issue Labels
- help wanted (21)
- enhancement (16)
- documentation (15)
- ECMWF (11)
- ๐ก Idea (8)
- bug (7)
- GFS (3)
- RAP (3)
- CFS (2)
- upstream (2)
- question (2)
- ๐ Canada (2)
- good first issue (2)
- accessors (2)
- NAM (1)
- FastHerbie (1)
- ๐งช CI/CD (1)
- inventory (1)
- RRFS (1)
- GEFS (1)
- NBM (1)
- wontfix (1)
- needs more info (1)
- ๐ searchString (1)
- windows (1)
Top Pull Request Labels
- enhancement (4)
- ECMWF (3)
- help wanted (1)
- HAFS (1)
- install (1)
- RRFS (1)
- GEFS (1)
Package metadata
- Total packages: 3
-
Total downloads:
- pypi: 56,966 last-month
- Total dependent packages: 3 (may contain duplicates)
- Total dependent repositories: 3 (may contain duplicates)
- Total versions: 28
- Total maintainers: 1
pypi.org: herbie-data
Download numerical weather prediction GRIB2 model data.
- Homepage: https://github.com/blaylockbk/Herbie
- Documentation: https://herbie.readthedocs.io/
- Licenses: MIT License Copyright (c) 2019-2024 Brian K. Blaylock Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
- Latest release: 2025.3.1 (published about 1 month ago)
- Last Synced: 2025-04-25T13:05:52.749Z (1 day ago)
- Versions: 21
- Dependent Packages: 3
- Dependent Repositories: 1
- Downloads: 56,468 Last month
-
Rankings:
- Stargazers count: 3.683%
- Forks count: 5.604%
- Downloads: 6.649%
- Dependent packages count: 7.31%
- Average: 9.067%
- Dependent repos count: 22.088%
- Maintainers (1)
pypi.org: hrrrb
Download model data (HRRR, RAP, GFS, NBM, etc.) from NOMADS, NOAA's Big Data Program partners (Amazon, Google, Microsoft), and the University of Utah Pando Archive System.
- Homepage:
- Documentation: https://blaylockbk.github.io/Herbie/_build/html/
- Licenses: MIT
- Latest release: 0.0.6 (published over 3 years ago)
- Last Synced: 2025-04-25T13:05:52.490Z (1 day ago)
- Versions: 6
- Dependent Packages: 0
- Dependent Repositories: 1
- Downloads: 498 Last month
-
Rankings:
- Stargazers count: 3.683%
- Forks count: 5.604%
- Dependent packages count: 7.31%
- Average: 17.182%
- Dependent repos count: 22.088%
- Downloads: 47.224%
- Maintainers (1)
conda-forge.org: herbie-data
Herbie is a python package that downloads recent and archived numerical weather prediction (NWP) model output from different cloud archive sources. Its most popular capability is to download HRRR model data. NWP data in GRIB2 format can be read with xarray+cfgrib. Much of this data is made available through the NOAA Open Data Dissemination (NODD) Program (formerly the Big Data Program) which has made weather data more accessible than ever before.
- Homepage: https://github.com/blaylockbk/Herbie
- Licenses: MIT
- Latest release: 2022.9.0.post1 (published over 2 years ago)
- Last Synced: 2025-04-01T02:10:15.489Z (26 days ago)
- Versions: 1
- Dependent Packages: 0
- Dependent Repositories: 1
-
Rankings:
- Dependent repos count: 24.103%
- Stargazers count: 24.852%
- Forks count: 26.733%
- Average: 31.807%
- Dependent packages count: 51.54%
Dependencies
- cfgrib >=0.9.9.1
- metpy >=1.3.0
- numpy >=1.22.3
- pandas >=1.4.1
- requests >=2.27.1
- toml >=0.10.2
- xarray >=2022.3.0
- cartopy *
- cfgrib *
- matplotlib *
- metpy *
- numpy *
- pandas *
- pygrib *
- toml *
- xarray *
- cartopy *
- cfgrib *
- metpy *
- numpy *
- pandas *
- pygrib *
- toml *
- xarray *
- actions/checkout v2 composite
- psf/black stable composite
- actions/checkout v3 composite
- actions/setup-python v3 composite
- actions/cache v3 composite
- actions/checkout v3 composite
- conda-incubator/setup-miniconda v2 composite
- actions/checkout v3 composite
- actions/setup-python v4 composite
- pytest <8 test
Score: 20.656892593735165