Intake-esm

An intake plugin for parsing an Earth System Model catalog and loading assets into xarray datasets.
https://github.com/intake/intake-esm

Category: Climate Change
Sub Category: Climate Data Access and Visualization

Keywords

cesm-lens climate-datasets cmip6 data-access data-catalog earth-system-model hacktoberfest intake pangeo

Keywords from Contributors

climate archiving climate-model climate-analysis compressed climate-science zarr observations catalogs intake-esm

Last synced: about 13 hours ago
JSON representation

Repository metadata

An intake plugin for parsing an Earth System Model (ESM) catalog and loading assets into xarray datasets.

Host: GitHub
URL: https://github.com/intake/intake-esm
Owner: intake
License: apache-2.0
Created: 2018-12-31T16:42:54.000Z (over 6 years ago)
Default Branch: main
Last Pushed: 2025-04-14T04:13:30.000Z (13 days ago)
Last Synced: 2025-04-21T11:07:50.231Z (6 days ago)
Topics: cesm-lens, climate-datasets, cmip6, data-access, data-catalog, earth-system-model, hacktoberfest, intake, pangeo
Language: Python
Homepage: https://intake-esm.readthedocs.io
Size: 11.7 MB
Stars: 149
Watchers: 14
Forks: 48
Open Issues: 36
Releases: 27
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Codeowners: .github/CODEOWNERS

Intake-esm

Intake-esm

Badges

CI
Docs
Package
License
Citation

Motivation

Computer simulations of the Earth’s climate and weather generate huge amounts of data.
These data are often persisted on HPC systems or in the cloud across multiple data
assets of a variety of formats (netCDF, zarr, etc...). Finding, investigating,
loading these data assets into compute-ready data containers costs time and effort.
The data user needs to know what data sets are available, the attributes describing
each data set, before loading a specific data set and analyzing it.

Finding, investigating, loading these assets into data array containers
such as xarray can be a daunting task due to the large number of files
a user may be interested in. Intake-esm aims to address these issues by
providing necessary functionality for searching, discovering, data access/loading.

Overview

intake-esm is a data cataloging utility built on top of intake, pandas, and xarray, and it's pretty awesome!

Opening an ESM catalog definition file: An Earth System Model (ESM) catalog file is a JSON file that conforms
to the ESM Collection Specification. When provided a link/path to an esm catalog file, intake-esm establishes
a link to a database (CSV file) that contains data assets locations and associated metadata
(i.e., which experiment, model, the come from). The catalog JSON file can be stored on a local filesystem
or can be hosted on a remote server.
```
In [1]: import intake

In [2]: import intake_esm

In [3]: cat_url = intake_esm.tutorial.get_url("google_cmip6")

In [4]: cat = intake.open_esm_datastore(cat_url)

In [5]: cat
Out[5]: <GOOGLE-CMIP6 catalog with 4 dataset(s) from 261 asset(s>
```

Search and Discovery: intake-esm provides functionality to execute queries against the catalog:

In [5]: cat_subset = cat.search(
   ...:     experiment_id=["historical", "ssp585"],
   ...:     table_id="Oyr",
   ...:     variable_id="o2",
   ...:     grid_label="gn",
   ...: )

In [6]: cat_subset
Out[6]: <GOOGLE-CMIP6 catalog with 4 dataset(s) from 261 asset(s)>

Access: when the user is satisfied with the results of their query, they can load data assets (netCDF and/or Zarr stores) into xarray datasets:


  In [7]: dset_dict = cat_subset.to_dataset_dict()

  --> The keys in the returned dictionary of datasets are constructed as follows:
          'activity_id.institution_id.source_id.experiment_id.table_id.grid_label'
  |███████████████████████████████████████████████████████████████| 100.00% [2/2 00:18<00:00]

See documentation for more information.

Installation

Intake-esm can be installed from PyPI with pip:

python -m pip install intake-esm

It is also available from conda-forge for conda installations:

conda install -c conda-forge intake-esm

Owner metadata

Name: Intake
Login: intake
Email: [email protected]
Kind: organization
Description: Taking the pain out of data access and distribution
Website: https://intake.readthedocs.io/en/latest/
Location:
Twitter:
Company:
Icon url: https://avatars.githubusercontent.com/u/1469464?v=4
Repositories: 29
Last ynced at: 2023-03-02T23:30:46.422Z
Profile URL: https://github.com/intake

GitHub Events

Total

Create event: 20
Release event: 1
Issues event: 15
Watch event: 11
Delete event: 15
Member event: 3
Issue comment event: 66
Push event: 60
Pull request review comment event: 2
Pull request review event: 12
Pull request event: 45
Fork event: 3

Last Year

Create event: 20
Release event: 1
Issues event: 15
Watch event: 11
Delete event: 15
Member event: 3
Issue comment event: 66
Push event: 60
Pull request review comment event: 2
Pull request review event: 12
Pull request event: 45
Fork event: 3

Committers metadata

Last synced: 4 days ago

Total Commits: 1,011
Total Committers: 25
Avg Commits per committer: 40.44
Development Distribution Score (DDS): 0.275

Commits in past year: 35
Committers in past year: 9
Avg Commits per committer in past year: 3.889
Development Distribution Score (DDS) in past year: 0.657

Name	Email	Commits
Anderson Banihirwe	a**e@u**u	733
mclong	m**g@u**u	77
pre-commit-ci[bot]	6****]	64
dependabot[bot]	4****]	46
Max Grover	m**x@g**m	19
bonnland	b**d@u**u	14
Julia Kent	4****t	13
Pascal Bourgault	b**l@o**a	12
Charles Turner	5****1	5
AS	a**g@m**e	4
Paul Branson	b**7@o**u	4
Dougie Squire	4****e	4
jbusecke	j**s@l**u	2
Trevor James Smith	1****e	2
Paul Branson	p**n@c**u	2
Aaron Spring	a****g	1
Hauke Schulz	4****s	1
Jared Lewis	j**d@j**z	1
Romain Beucher	r**r@a**u	1
RondeauG	3****G	1
Sadie L. Bartholomew	s**w@n**k	1
Sebastián Blanco	s**g@e**m	1
Joseph Hamman	j**n@u**u	1
Tobias Kölling	t**i@d**e	1
garciampred	9****d	1

Committer domains:

Issue and Pull Request metadata

Last synced: 1 day ago

Total issues: 228
Total pull requests: 454
Average time to close issues: 3 months
Average time to close pull requests: 5 days
Total issue authors: 64
Total pull request authors: 28
Average comments per issue: 3.73
Average comments per pull request: 1.02
Merged pull request: 427
Bot issues: 0
Bot pull requests: 114

Past year issues: 12
Past year pull requests: 40
Past year average time to close issues: about 1 month
Past year average time to close pull requests: 6 days
Past year issue authors: 8
Past year pull request authors: 10
Past year average comments per issue: 3.75
Past year average comments per pull request: 0.63
Past year merged pull request: 36
Past year bot issues: 0
Past year bot pull requests: 22

More stats: https://issues.ecosyste.ms/repositories/lookup?url=https://github.com/intake/intake-esm

Top Issue Authors

andersy005 (42)
jbusecke (23)
matt-long (19)
aulemahal (17)
ahuang11 (8)
naomi-henderson (7)
aaronspring (7)
mgrover1 (7)
dougiesquire (6)
wachsylon (6)
charles-turner-1 (5)
fanchic (4)
aradhakrishnanGFDL (4)
jukent (4)
dcherian (3)

Top Pull Request Authors

andersy005 (238)
pre-commit-ci[bot] (66)
dependabot[bot] (48)
mgrover1 (20)
matt-long (17)
jukent (14)
aulemahal (13)
charles-turner-1 (7)
dougiesquire (5)
aaronspring (3)
bonnland (3)
Zeitsperre (2)
sadielbartholomew (2)
jbusecke (2)
d70-t (1)

Top Issue Labels

enhancement (41)
bug (33)
usage question (28)
documentation (7)
discuss (3)
good first issue (3)
needs triage (3)
feature (2)
awaiting more information (2)
upstream issue (2)
help wanted (1)

Top Pull Request Labels

dependencies (57)
maintenance (51)
enhancement (36)
CI (26)
documentation (22)
usage question (14)
bug-fix (12)
feature (6)
internal-change (5)

Package metadata

Total packages: 2
Total downloads:
- pypi: 14,974 last-month
Total docker downloads: 1,563
Total dependent packages: 18 (may contain duplicates)
Total dependent repositories: 86 (may contain duplicates)
Total versions: 46
Total maintainers: 3

pypi.org: intake-esm

An intake plugin for parsing an Earth System Model (ESM) catalog and loading netCDF files and/or Zarr stores into Xarray datasets.

Homepage: https://intake-esm.readthedocs.io
Documentation: https://intake-esm.readthedocs.io
Licenses: Apache Software License 2.0
Latest release: 2025.2.3 (published 3 months ago)
Last Synced: 2025-04-25T13:05:31.974Z (1 day ago)
Versions: 27
Dependent Packages: 15
Dependent Repositories: 22
Downloads: 14,974 Last month
Docker Downloads: 1,563
Rankings:
- Dependent packages count: 1.596%
- Downloads: 1.872%
- Docker downloads count: 2.319%
- Dependent repos count: 3.12%
- Average: 3.631%
- Forks count: 6.29%
- Stargazers count: 6.588%
Maintainers (3)

conda-forge.org: intake-esm

An intake plugin for parsing an Earth System Model (ESM) collection/catalog and loading assets (netCDF files and/or Zarr stores) into xarray data sets.

Homepage: https://github.com/intake/intake-esm
Licenses: Apache-2.0
Latest release: 2022.9.18 (published over 2 years ago)
Last Synced: 2025-04-25T13:05:45.443Z (1 day ago)
Versions: 19
Dependent Packages: 3
Dependent Repositories: 64
Rankings:
- Dependent repos count: 4.392%
- Dependent packages count: 15.638%
- Average: 20.327%
- Forks count: 28.355%
- Stargazers count: 32.922%

Dependencies

requirements.txt pypi

dask >=2021.9
fastprogress >=1.0.0
fsspec >=2021.7.0
intake >=0.6.5
netCDF4 >=1.5.5
pydantic >=1.8.2
requests >=2.24.0
xarray >=0.19,
xcollection *
zarr >=2.5

.github/workflows/ci.yaml actions

actions/checkout v3 composite
codecov/codecov-action v3.1.1 composite
mamba-org/provision-with-micromamba main composite

.github/workflows/pypi.yml actions

actions/checkout v3 composite
actions/setup-python v4 composite
pypa/gh-action-pypi-publish v1.6.4 composite

pyproject.toml pypi

ci/environment.yml conda

cftime
codecov
fastprogress >=1.0.0
fsspec >=2022.11.0
gcsfs >=2022.11.0
h5netcdf >=0.8.1
intake <2.0
ipython
matplotlib
netcdf4 >=1.5.5
pandas >=2.1.0
pip
pooch
pre-commit
pydantic >=2.0
pydap
pytest
pytest-cov
pytest-mock
pytest-sugar
pytest-xdist
s3fs >=2022.11.0
scipy
xarray >=2022.06
xarray-datatree
zarr >=2.12

Score: 18.15963735674229

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Open Sustainable Technology

Intake-esm

Keywords

Keywords from Contributors

Repository metadata

README.md

Intake-esm

Badges

Motivation

Overview

Installation

Owner metadata

GitHub Events

Total

Last Year

Committers metadata

Committer domains:

Issue and Pull Request metadata

Top Issue Authors

Top Pull Request Authors

Top Issue Labels

Top Pull Request Labels

Package metadata

pypi.org: intake-esm

conda-forge.org: intake-esm

Dependencies