h5netcdf
A Python interface for the netCDF4 file format that reads and writes local or remote HDF5 files directly via h5py or h5pyd, without relying on the Unidata netCDF library.
https://github.com/h5netcdf/h5netcdf
Category: Climate Change
Sub Category: Climate Data Standards
Keywords
h5py hdf5 netcdf python
Keywords from Contributors
pydata closember measur qt profile archiving flexible alignment fish observations
Last synced: about 3 hours ago
JSON representation
Repository metadata
Pythonic interface to netCDF4 via h5py
- Host: GitHub
- URL: https://github.com/h5netcdf/h5netcdf
- Owner: h5netcdf
- License: bsd-3-clause
- Created: 2015-04-07T18:44:42.000Z (about 10 years ago)
- Default Branch: main
- Last Pushed: 2025-03-07T14:51:02.000Z (about 2 months ago)
- Last Synced: 2025-04-17T22:44:35.726Z (13 days ago)
- Topics: h5py, hdf5, netcdf, python
- Language: Python
- Homepage: https://h5netcdf.org
- Size: 6.99 MB
- Stars: 198
- Watchers: 5
- Forks: 36
- Open Issues: 18
- Releases: 18
-
Metadata Files:
- Readme: README.rst
- Changelog: CHANGELOG.rst
- License: LICENSE
- Authors: AUTHORS.txt
README.rst
h5netcdf ======== .. image:: https://github.com/h5netcdf/h5netcdf/workflows/CI/badge.svg :target: https://github.com/h5netcdf/h5netcdf/actions .. image:: https://badge.fury.io/py/h5netcdf.svg :target: https://pypi.org/project/h5netcdf/ .. image:: https://github.com/h5netcdf/h5netcdf/actions/workflows/pages/pages-build-deployment/badge.svg?branch=gh-pages :target: https://h5netcdf.github.io/h5netcdf/ A Python interface for the `netCDF4`_ file-format that reads and writes local or remote HDF5 files directly via `h5py`_ or `h5pyd`_, without relying on the Unidata netCDF library. .. _netCDF4: https://docs.unidata.ucar.edu/netcdf-c/current/file_format_specifications.html#netcdf_4_spec .. _h5py: https://www.h5py.org/ .. _h5pyd: https://github.com/HDFGroup/h5pyd .. why-h5netcdf Why h5netcdf? ------------- - It has one less binary dependency (netCDF C). If you already have h5py installed, reading netCDF4 with h5netcdf may be much easier than installing netCDF4-Python. - We've seen occasional reports of better performance with h5py than netCDF4-python, though in many cases performance is identical. For `one workflow`_, h5netcdf was reported to be almost **4x faster** than `netCDF4-python`_. - Anecdotally, HDF5 users seem to be unexcited about switching to netCDF -- hopefully this will convince them that netCDF4 is actually quite sane! - Finally, side-stepping the netCDF C library (and Cython bindings to it) gives us an easier way to identify the source of performance issues and bugs in the netCDF libraries/specification. .. _one workflow: https://github.com/Unidata/netcdf4-python/issues/390#issuecomment-93864839 .. _xarray: https://github.com/pydata/xarray/ Install ------- Ensure you have a recent version of h5py installed (I recommend using `conda`_ or the community effort `conda-forge`_). At least version 3.0 is required. Then:: $ pip install h5netcdf Or if you are already using conda:: $ conda install h5netcdf Note: From version 1.2. h5netcdf tries to align with a `nep29`_-like support policy with regard to it's upstream dependencies. .. _conda: https://conda.io/ .. _conda-forge: https://conda-forge.org/ .. _nep29: https://numpy.org/neps/nep-0029-deprecation_policy.html Usage ----- h5netcdf has two APIs, a new API and a legacy API. Both interfaces currently reproduce most of the features of the netCDF interface, with the notable exception of support for operations that rename or delete existing objects. We simply haven't gotten around to implementing this yet. Patches would be very welcome. New API ~~~~~~~ The new API supports direct hierarchical access of variables and groups. Its design is an adaptation of h5py to the netCDF data model. For example: .. code-block:: python import h5netcdf import numpy as np with h5netcdf.File("mydata.nc", "w") as f: # set dimensions with a dictionary f.dimensions = {"x": 5} # and update them with a dict-like interface # f.dimensions['x'] = 5 # f.dimensions.update({'x': 5}) v = f.create_variable("hello", ("x",), float) v[:] = np.ones(5) # you don't need to create groups first # you also don't need to create dimensions first if you supply data # with the new variable v = f.create_variable("/grouped/data", ("y",), data=np.arange(10)) # access and modify attributes with a dict-like interface v.attrs["foo"] = "bar" # you can access variables and groups directly using a hierarchical # keys like h5py print(f["/grouped/data"]) # add an unlimited dimension f.dimensions["z"] = None # explicitly resize a dimension and all variables using it f.resize_dimension("z", 3) Notes: - Automatic resizing of unlimited dimensions with array indexing is not available. - Dimensions need to be manually resized with ``Group.resize_dimension(dimension, size)``. - Arrays are returned padded with ``fillvalue`` (taken from underlying hdf5 dataset) up to current size of variable's dimensions. The behaviour is equivalent to netCDF4-python's ``Dataset.set_auto_mask(False)``. Legacy API ~~~~~~~~~~ The legacy API is designed for compatibility with `netCDF4-python`_. To use it, import ``h5netcdf.legacyapi``: .. _netCDF4-python: https://github.com/Unidata/netcdf4-python .. code-block:: python import h5netcdf.legacyapi as netCDF4 # everything here would also work with this instead: # import netCDF4 import numpy as np with netCDF4.Dataset("mydata.nc", "w") as ds: ds.createDimension("x", 5) v = ds.createVariable("hello", float, ("x",)) v[:] = np.ones(5) g = ds.createGroup("grouped") g.createDimension("y", 10) g.createVariable("data", "i8", ("y",)) v = g["data"] v[:] = np.arange(10) v.foo = "bar" print(ds.groups["grouped"].variables["data"]) The legacy API is designed to be easy to try-out for netCDF4-python users, but it is not an exact match. Here is an incomplete list of functionality we don't include: - Utility functions ``chartostring``, ``num2date``, etc., that are not directly necessary for writing netCDF files. - h5netcdf variables do not support automatic masking or scaling (e.g., of values matching the ``_FillValue`` attribute). We prefer to leave this functionality to client libraries (e.g., `xarray`_), which can implement their exact desired scaling behavior. Nevertheless arrays are returned padded with ``fillvalue`` (taken from underlying hdf5 dataset) up to current size of variable's dimensions. The behaviour is equivalent to netCDF4-python's ``Dataset.set_auto_mask(False)``. .. _invalid netcdf: Invalid netCDF files ~~~~~~~~~~~~~~~~~~~~ h5py implements some features that do not (yet) result in valid netCDF files: - Data types: - Booleans - Reference types - Arbitrary filters: - Scale-offset filters By default [#]_, h5netcdf will not allow writing files using any of these features, as files with such features are not readable by other netCDF tools. However, these are still valid HDF5 files. If you don't care about netCDF compatibility, you can use these features by setting ``invalid_netcdf=True`` when creating a file: .. code-block:: python # avoid the .nc extension for non-netcdf files f = h5netcdf.File("mydata.h5", invalid_netcdf=True) ... # works with the legacy API, too, though compression options are not exposed ds = h5netcdf.legacyapi.Dataset("mydata.h5", invalid_netcdf=True) ... In such cases the `_NCProperties` attribute will not be saved to the file or be removed from an existing file. A warning will be issued if the file has `.nc`-extension. .. rubric:: Footnotes .. [#] h5netcdf we will raise ``h5netcdf.CompatibilityError``. Decoding variable length strings ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ h5py 3.0 introduced `new behavior`_ for handling variable length string. Instead of being automatically decoded with UTF-8 into NumPy arrays of ``str``, they are required as arrays of ``bytes``. The legacy API preserves the old behavior of h5py (which matches netCDF4), and automatically decodes strings. The new API matches h5py behavior. Explicitly set ``decode_vlen_strings=True`` in the ``h5netcdf.File`` constructor to opt-in to automatic decoding. .. _new behavior: https://docs.h5py.org/en/stable/strings.html .. _phony dims: Datasets with missing dimension scales ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ By default [#]_ h5netcdf raises a ``ValueError`` if variables with no dimension scale associated with one of their axes are accessed. You can set ``phony_dims='sort'`` when opening a file to let h5netcdf invent phony dimensions according to `netCDF`_ behaviour. .. code-block:: python # mimic netCDF-behaviour for non-netcdf files f = h5netcdf.File("mydata.h5", mode="r", phony_dims="sort") ... Note, that this iterates once over the whole group-hierarchy. This has affects on performance in case you rely on laziness of group access. You can set ``phony_dims='access'`` instead to defer phony dimension creation to group access time. The created phony dimension naming will differ from `netCDF`_ behaviour. .. code-block:: python f = h5netcdf.File("mydata.h5", mode="r", phony_dims="access") ... .. rubric:: Footnotes .. [#] Keyword default setting ``phony_dims=None`` for backwards compatibility. .. _netCDF: https://docs.unidata.ucar.edu/netcdf-c/current/interoperability_hdf5.html Track Order ~~~~~~~~~~~ As of h5netcdf 1.1.0, if h5py 3.7.0 or greater is detected, the ``track_order`` parameter is set to ``True`` enabling `order tracking`_ for newly created netCDF4 files. This helps ensure that files created with the h5netcdf library can be modified by the netCDF4-c and netCDF4-python implementation used in other software stacks. Since this change should be transparent to most users, it was made without deprecation. Since track_order is set at creation time, any dataset that was created with ``track_order=False`` (h5netcdf version 1.0.2 and older except for 0.13.0) will continue to opened with order tracker disabled. The following describes the behavior of h5netcdf with respect to order tracking for a few key versions: - Version 0.12.0 and earlier, the ``track_order`` parameter`order was missing and thus order tracking was implicitely set to ``False``. - Version 0.13.0 enabled order tracking by setting the parameter ``track_order`` to ``True`` by default without deprecation. - Versions 0.13.1 to 1.0.2 set ``track_order`` to ``False`` due to a bug in a core dependency of h5netcdf, h5py `upstream bug`_ which was resolved in h5py 3.7.0 with the help of the h5netcdf team. - In version 1.1.0, if h5py 3.7.0 or above is detected, the ``track_order`` parameter is set to ``True`` by default. .. _order tracking: https://docs.unidata.ucar.edu/netcdf-c/current/file_format_specifications.html#creation_order .. _upstream bug: https://github.com/h5netcdf/h5netcdf/issues/136 .. _[*]: https://github.com/h5netcdf/h5netcdf/issues/128 .. changelog Changelog --------- `Changelog`_ .. _Changelog: https://github.com/h5netcdf/h5netcdf/blob/main/CHANGELOG.rst .. license License ------- `3-clause BSD`_ .. _3-clause BSD: https://github.com/h5netcdf/h5netcdf/blob/main/LICENSE
Owner metadata
- Name: h5netcdf
- Login: h5netcdf
- Email:
- Kind: organization
- Description:
- Website:
- Location:
- Twitter:
- Company:
- Icon url: https://avatars.githubusercontent.com/u/74427192?v=4
- Repositories: 1
- Last ynced at: 2023-03-01T05:25:22.286Z
- Profile URL: https://github.com/h5netcdf
GitHub Events
Total
- Create event: 4
- Release event: 4
- Issues event: 9
- Watch event: 13
- Delete event: 1
- Issue comment event: 70
- Push event: 13
- Pull request review comment event: 4
- Pull request event: 27
- Pull request review event: 7
- Fork event: 2
Last Year
- Create event: 4
- Release event: 4
- Issues event: 9
- Watch event: 13
- Delete event: 1
- Issue comment event: 70
- Push event: 13
- Pull request review comment event: 4
- Pull request event: 27
- Pull request review event: 7
- Fork event: 2
Committers metadata
Last synced: 8 days ago
Total Commits: 263
Total Committers: 21
Avg Commits per committer: 12.524
Development Distribution Score (DDS): 0.654
Commits in past year: 18
Committers in past year: 6
Avg Commits per committer in past year: 3.0
Development Distribution Score (DDS) in past year: 0.389
Name | Commits | |
---|---|---|
Kai Mühlbauer | k****r@u****e | 91 |
Stephan Hoyer | s****r@c****m | 87 |
Aleksandar Jelenak | a****k@g****e | 20 |
Lion Krischer | l****r@g****m | 16 |
Mark Harfouche | m****e@g****m | 11 |
Aleksandar Jelenak | a****k@h****g | 7 |
Scott Henderson | s****q@g****m | 5 |
dependabot[bot] | 4****] | 5 |
Martin Raspaud | m****d@s****e | 3 |
Brett Naul | b****l@g****m | 2 |
Frederic Laliberte | l****c@g****m | 2 |
Ghislain Antony Vaillant | g****l | 2 |
Rickard Holmberg | r****g@n****m | 2 |
Dion Häfner | d****r@n****k | 2 |
Tom Augspurger | t****r@m****m | 2 |
Drew Parsons | d****s@e****m | 1 |
Ezequiel Cimadevilla Alvarez | e****a@u****s | 1 |
John Readey | j****y@h****g | 1 |
Ryan Grout | g****r | 1 |
Thomas Kluyver | t****l@g****m | 1 |
paugier | p****r@u****r | 1 |
Committer domains:
- hdfgroup.org: 2
- univ-grenoble-alpes.fr: 1
- unican.es: 1
- emerall.com: 1
- microsoft.com: 1
- nbi.ku.dk: 1
- novatronfusion.com: 1
- smhi.se: 1
- gharial.home: 1
- climate.com: 1
- uni-bonn.de: 1
Issue and Pull Request metadata
Last synced: 1 day ago
Total issues: 115
Total pull requests: 144
Average time to close issues: 12 months
Average time to close pull requests: 24 days
Total issue authors: 61
Total pull request authors: 26
Average comments per issue: 5.09
Average comments per pull request: 3.82
Merged pull request: 123
Bot issues: 0
Bot pull requests: 6
Past year issues: 9
Past year pull requests: 18
Past year average time to close issues: 2 months
Past year average time to close pull requests: 9 days
Past year issue authors: 8
Past year pull request authors: 6
Past year average comments per issue: 3.78
Past year average comments per pull request: 2.28
Past year merged pull request: 16
Past year bot issues: 0
Past year bot pull requests: 2
Top Issue Authors
- kmuehlbauer (19)
- shoyer (14)
- hmaarrfk (8)
- mangecoeur (4)
- dionhaefner (3)
- drew-parsons (3)
- mraspaud (2)
- Hellseher (2)
- laliberte (2)
- nschloe (2)
- bnlawrence (2)
- paugier (2)
- gerritholl (2)
- fsvenson (2)
- ghisvail (2)
Top Pull Request Authors
- kmuehlbauer (79)
- hmaarrfk (17)
- shoyer (11)
- dependabot[bot] (6)
- TomAugspurger (3)
- rho-novatron (2)
- ajelenak (2)
- ghost (2)
- ghisvail (2)
- laliberte (2)
- dionhaefner (2)
- zequihg50 (2)
- pnuu (1)
- basnijholt (1)
- carlosal1015 (1)
Top Issue Labels
- bug (10)
- enhancement (9)
- wontfix (3)
- upstream-HDF5 (2)
- needs info (1)
Top Pull Request Labels
- dependencies (6)
- bugfix (4)
Package metadata
- Total packages: 4
-
Total downloads:
- pypi: 1,302,433 last-month
- Total docker downloads: 310,858
- Total dependent packages: 177 (may contain duplicates)
- Total dependent repositories: 786 (may contain duplicates)
- Total versions: 78
- Total maintainers: 3
pypi.org: h5netcdf
netCDF4 via h5py
- Homepage:
- Documentation: https://h5netcdf.readthedocs.io/
- Licenses: Copyright (c) 2015, h5netcdf developers All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- Latest release: 1.6.1 (published about 2 months ago)
- Last Synced: 2025-04-29T15:05:12.800Z (1 day ago)
- Versions: 45
- Dependent Packages: 142
- Dependent Repositories: 430
- Downloads: 1,302,433 Last month
- Docker Downloads: 310,858
-
Rankings:
- Dependent packages count: 0.151%
- Downloads: 0.491%
- Average: 0.685%
- Dependent repos count: 0.694%
- Docker downloads count: 1.404%
- Maintainers (2)
spack.io: py-h5netcdf
A Python interface for the netCDF4 file-format that reads and writes local or remote HDF5 files directly via h5py or h5pyd, without relying on the Unidata netCDF library.
- Homepage: https://github.com/h5netcdf/h5netcdf
- Licenses: []
- Latest release: 1.3.0 (published about 1 year ago)
- Last Synced: 2024-11-29T10:27:41.070Z (5 months ago)
- Versions: 2
- Dependent Packages: 1
- Dependent Repositories: 0
-
Rankings:
- Dependent repos count: 0.0%
- Average: 15.68%
- Stargazers count: 15.826%
- Forks count: 18.827%
- Dependent packages count: 28.067%
- Maintainers (1)
conda-forge.org: h5netcdf
- Homepage: https://github.com/h5netcdf/h5netcdf
- Licenses: BSD-3-Clause
- Latest release: 1.0.2 (published over 2 years ago)
- Last Synced: 2025-04-02T02:58:34.777Z (29 days ago)
- Versions: 30
- Dependent Packages: 33
- Dependent Repositories: 178
-
Rankings:
- Dependent packages count: 2.061%
- Dependent repos count: 2.605%
- Average: 16.26%
- Stargazers count: 28.565%
- Forks count: 31.809%
anaconda.org: h5netcdf
A Python interface for the netCDF4 file-format that reads and writes local or remote HDF5 files directly via h5py or h5pyd, without relying on the Unidata netCDF library.
- Homepage: https://github.com/h5netcdf/h5netcdf
- Licenses: BSD-3-Clause
- Latest release: 1.2.0 (published almost 2 years ago)
- Last Synced: 2025-04-29T15:05:13.079Z (1 day ago)
- Versions: 1
- Dependent Packages: 1
- Dependent Repositories: 178
-
Rankings:
- Dependent repos count: 14.969%
- Average: 34.824%
- Stargazers count: 40.215%
- Dependent packages count: 40.951%
- Forks count: 43.16%
Dependencies
- actions/checkout v2 composite
- actions/checkout v3 composite
- actions/setup-python v3 composite
- actions/upload-artifact v1 composite
- mamba-org/provision-with-micromamba main composite
- peaceiris/actions-gh-pages v3 composite
Score: 22.71421902416947