icechunk

An open-source transactional storage engine for tensor / ND-array data designed for use on cloud object storage.
https://github.com/earth-mover/icechunk

Category: Climate Change
Sub Category: Climate Data Standards

Keywords

cloud database xarray zarr

Keywords from Contributors

compressed ndimensional-arrays pangeo pydata earth-observation open-science closember gdal cloud-computing climate

Last synced: about 12 hours ago
JSON representation

Repository metadata

Open-source, cloud-native transactional tensor storage engine

Host: GitHub
URL: https://github.com/earth-mover/icechunk
Owner: earth-mover
License: apache-2.0
Created: 2024-07-30T02:48:26.000Z (about 2 years ago)
Default Branch: main
Last Pushed: 2026-09-24T20:11:03.000Z (2 days ago)
Last Synced: 2026-09-25T08:31:38.960Z (1 day ago)
Topics: cloud, database, xarray, zarr
Language: Rust
Homepage: https://icechunk.io
Size: 18.8 MB
Stars: 685
Watchers: 20
Forks: 87
Open Issues: 271
Releases: 76
Metadata Files:
- Readme: README.md
- Changelog: Changelog.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md

Icechunk

[!TIP]
Icechunk 2.0 is released! Better consistency, performance, and reliability for tensor storage

Icechunk is an open-source (Apache 2.0), transactional storage engine for tensor / ND-array data designed for use on cloud object storage.
Icechunk works together with Zarr, augmenting the Zarr core data model with features
that enhance performance, collaboration, and safety in a cloud-computing context.

Documentation and Resources

This page: a general overview of the project's goals and components.
Icechunk Launch Blog Post
Frequently Asked Questions
Documentation for Icechunk Python, the main user-facing
library
Documentation for the Icechunk Rust Crate
The Contributor Guide
The Icechunk Spec

Crate Structure

The Rust workspace is organized into layered crates:

graph TD
    python[icechunk-python] --> core[icechunk]
    core --> arrow[icechunk-arrow-object-store]
    core --> s3[icechunk-s3 *optional*]
    core --> storage[icechunk-storage]
    core --> format[icechunk-format]
    arrow --> storage
    s3 --> storage
    storage --> types[icechunk-types]
    format --> types

Crate	Description
icechunk-macros	Procedural macro helpers for tests and internal use
icechunk-types	Shared foundational types (`Path`, `ETag`, `Move`, error wrappers) used across all crates
icechunk-format	Binary format types and serialization (snapshots, manifests, transaction logs, repo info)
icechunk-storage	Storage trait definitions and common storage utilities
icechunk-arrow-object-store	Storage backend using Apache Arrow's `object_store` (in-memory, local, GCS, Azure, etc.)
icechunk-s3	Native AWS S3 storage backend (optional feature)
icechunk	Core storage engine: transactions, version control, repositories
icechunk-python	PyO3 bindings exposing the engine to Python

Icechunk Overview

Let's break down what "transactional storage engine for Zarr" actually means:

Zarr is an open source specification for the storage of multidimensional array (a.k.a. tensor) data.
Zarr defines the metadata for describing arrays (shape, dtype, etc.) and the way these arrays are chunked, compressed, and converted to raw bytes for storage. Zarr can store its data in any key-value store.
There are many different implementations of Zarr in different languages. Right now, Icechunk only supports
Zarr Python.
If you're interested in implementing Icechunk support, please open an issue so we can help you.
Storage engine - Icechunk exposes a key-value interface to Zarr and manages all of the actual I/O for getting, setting, and updating both metadata and chunk data in cloud object storage.
Zarr libraries don't have to know exactly how icechunk works under the hood in order to use it.
Transactional - The key improvement that Icechunk brings on top of regular Zarr is to provide consistent serializable isolation between transactions.
This means that Icechunk data is safe to read and write in parallel from multiple uncoordinated processes.
This allows Zarr to be used more like a database.

The core entity in Icechunk is a repository or repo.
A repo is defined as a Zarr hierarchy containing one or more Arrays and Groups, and a repo functions as a
self-contained Zarr Store.
The most common scenario is for an Icechunk repo to contain a single Zarr group with multiple arrays, each corresponding to different physical variables but sharing common spatiotemporal coordinates.
However, formally a repo can be any valid Zarr hierarchy, from a single Array to a deeply nested structure of Groups and Arrays.
Users of Icechunk should aim to scope their repos only to related arrays and groups that require consistent transactional updates.

Icechunk supports the following core requirements:

Object storage - the format is designed around the consistency features and performance characteristics available in modern cloud object storage. No external database or catalog is required to maintain a repo.
(It also works with file storage.)
Serializable isolation - Reads are isolated from concurrent writes and always use a committed snapshot of a repo. Writes are committed atomically and are never partially visible. No locks are required for reading.
Time travel - Previous snapshots of a repo remain accessible after new ones have been written.
Data version control - Repos support both tags (immutable references to snapshots) and branches (mutable references to snapshots).
Chunk shardings - Chunk storage is decoupled from specific file names. Multiple chunks can be packed into a single object (sharding).
Chunk references - Zarr-compatible chunks within other file formats (e.g. HDF5, NetCDF) can be referenced.
Schema evolution - Arrays and Groups can be added, renamed, and removed from the hierarchy with minimal overhead.

Key Concepts

Groups, Arrays, and Chunks

Icechunk is designed around the Zarr data model, widely used in scientific computing, data science, and AI / ML.
(The Zarr high-level data model is effectively the same as HDF5.)
The core data structure in this data model is the array.
Arrays have two fundamental properties:

shape - a tuple of integers which specify the dimensions of each axis of the array. A 10 x 10 square array would have shape (10, 10)
data type - a specification of what type of data is found in each element, e.g. integer, float, etc. Different data types have different precision (e.g. 16-bit integer, 64-bit float, etc.)

In Zarr / Icechunk, arrays are split into chunks.
A chunk is the minimum unit of data that must be read / written from storage, and thus choices about chunking have strong implications for performance.
Zarr leaves this completely up to the user.
Chunk shape should be chosen based on the anticipated data access pattern for each array.
An Icechunk array is not bounded by an individual file and is effectively unlimited in size.

For further organization of data, Icechunk supports groups within a single repo.
Group are like folders which contain multiple arrays and or other groups.
Groups enable data to be organized into hierarchical trees.
A common usage pattern is to store multiple arrays in a group representing a NetCDF-style dataset.

Arbitrary JSON-style key-value metadata can be attached to both arrays and groups.

Snapshots

Every update to an Icechunk store creates a new snapshot with a unique ID.
Icechunk users must organize their updates into groups of related operations called transactions.
For example, appending a new time slice to multiple arrays should be done as a single transaction, comprising the following steps

Update the array metadata to resize the array to accommodate the new elements.
Write new chunks for each array in the group.

While the transaction is in progress, none of these changes will be visible to other users of the store.
Once the transaction is committed, a new snapshot is generated.
Readers can only see and use committed snapshots.

Branches and Tags

Additionally, snapshots occur in a specific linear (i.e. serializable) order within a branch.
A branch is a mutable reference to a snapshot--a pointer that maps the branch name to a snapshot ID.
The default branch is main.
Every commit to the main branch updates this reference.
Icechunk's design protects against the race condition in which two uncoordinated sessions attempt to update the branch at the same time; only one can succeed.

Icechunk also defines tags--immutable references to snapshot.
Tags are appropriate for publishing specific releases of a repository or for any application which requires a persistent, immutable identifier to the store state.

Chunk References

Chunk references are "pointers" to chunks that exist in other files--HDF5, NetCDF, GRIB, etc.
Icechunk can store these references alongside native Zarr chunks as "virtual datasets".
You can then update these virtual datasets incrementally (overwrite chunks, change metadata, etc.) without touching the underling files.

How Does It Work?

!!! Note:
For a more detailed explanation, have a look at the Icechunk spec.

Zarr itself works by storing both metadata and chunk data into a abstract store according to a specified system of "keys".
For example, a 2D Zarr array called myarray, within a group called mygroup, would generate the following keys:

mygroup/zarr.json
mygroup/myarray/zarr.json
mygroup/myarray/c/0/0
mygroup/myarray/c/0/1

In standard regular Zarr stores, these key map directly to filenames in a filesystem or object keys in an object storage system.
When writing data, a Zarr implementation will create these keys and populate them with data.
When modifying existing arrays or groups, a Zarr implementation will potentially overwrite existing keys with new data.

This is generally not a problem, as long there is only one person or process coordinating access to the data.
However, when multiple uncoordinated readers and writers attempt to access the same Zarr data at the same time, various consistency problems emerge.
These consistency problems can occur in both file storage and object storage; they are particularly severe in a cloud setting where Zarr is being used as an active store for data that are frequently changed while also being read.

With Icechunk, we keep the same core Zarr data model, but add a layer of indirection between the Zarr keys and the on-disk storage.
The Icechunk library translates between the Zarr keys and the actual on-disk data given the particular context of the user's state.
Icechunk defines a series of interconnected metadata and data files that together enable efficient isolated reading and writing of metadata and chunks.
Once written, these files are immutable.
Icechunk keeps track of every single chunk explicitly in a "chunk manifest".

flowchart TD
    zarr-python[Zarr Library] <-- key / value--> icechunk[Icechunk Library]
    icechunk <-- data / metadata files --> storage[(Object Storage)]

Owner metadata

Name: Earthmover
Login: earth-mover
Email: hello@earthmover.io
Kind: organization
Description: The cloud platform for scientific data teams
Website: https://earthmover.io/
Location: United States of America
Twitter: EarthmoverHQ
Company:
Icon url: https://avatars.githubusercontent.com/u/95875797?v=4
Repositories: 34
Last ynced at: 2026-09-25T13:54:08.492Z
Profile URL: https://github.com/earth-mover

GitHub Events

Total

Create event: 797
Commit comment event: 2
Release event: 37
Delete event: 724
Member event: 1
Pull request event: 1132
Fork event: 31
Discussion event: 2
Issues event: 785
Watch event: 458
Issue comment event: 1167
Push event: 3061
Gollum event: 2
Pull request review comment event: 1400
Pull request review event: 1684

Last Year

Create event: 353
Commit comment event: 2
Release event: 7
Delete event: 347
Pull request event: 243
Fork event: 4
Discussion event: 2
Issues event: 244
Watch event: 49
Issue comment event: 286
Push event: 1205
Pull request review event: 347
Pull request review comment event: 297

Committers metadata

Last synced: 1 day ago

Total Commits: 1,473
Total Committers: 57
Avg Commits per committer: 25.842
Development Distribution Score (DDS): 0.578

Commits in past year: 619
Committers in past year: 36
Avg Commits per committer in past year: 17.194
Development Distribution Score (DDS) in past year: 0.75

Name	Email	Commits
Sebastián Galkin	c**e@a**m	621
Matthew Iannucci	m**w@e**o	162
Deepak Cherian	d****n	154
dependabot[bot]	4****]	108
Ian Hunt-Isaak	i**k@g**m	99
Luiz Irber	l**z@e**o	97
Tom Nicholas	t**m@e**o	63
Ryan Abernathey	r**y@g**m	23
Joe Hamman	j**e@e**o	23
Orestis Herodotou	d****o	14
samantha-earthmover	s**a@e**o	14
Orestis Herodotou	o**s@e**o	8
Julius Busecke	1****e	8
Max Jones	1****s	6
jnotjay	8****y	4
Lindsey Nield	l**y@e**o	4
Aimee Barciauskas	a**e@d**g	4
Alex Kerney	a**k@m**m	3
Callum Rollo	c**o@o**m	3
Daniel Jahn (dahn)	d**n@g**m	3
Eitan Barylko	6****o	3
Eva Pace	e**i@g**m	3
Yaroslav Halchenko	d**n@o**m	3
Kyle Barron	k**e@d**g	3
Alfonso Ladino	a**r@u**o	2
David Brochart	d**t@g**m	2
Florian Jetter	f****r	2
Michael Sumner	m**r@g**m	2
Nick Hodgskin	3****o	2
Raphael Hagen	n**n@g**m	2
and 27 more...

Committer domains:

Issue and Pull Request metadata

Last synced: 1 day ago

Total issues: 475
Total pull requests: 989
Average time to close issues: about 1 month
Average time to close pull requests: 5 days
Total issue authors: 95
Total pull request authors: 60
Average comments per issue: 1.48
Average comments per pull request: 0.7
Merged pull request: 669
Bot issues: 33
Bot pull requests: 155

Past year issues: 119
Past year pull requests: 300
Past year average time to close issues: about 1 month
Past year average time to close pull requests: 5 days
Past year issue authors: 41
Past year pull request authors: 33
Past year average comments per issue: 2.13
Past year average comments per pull request: 0.96
Past year merged pull request: 188
Past year bot issues: 19
Past year bot pull requests: 79

More stats: https://issues.ecosyste.ms/repositories/lookup?url=https://github.com/earth-mover/icechunk

Top Issue Authors

paraseba (112)
dcherian (74)
github-actions[bot] (33)
mpiannucci (33)
ianhi (33)
TomNicholas (31)
rabernat (21)
jhamman (15)
jbusecke (8)
jbms (7)
samantha-earthmover (4)
li-em (3)
tomwhite (3)
lindseynield (3)
DahnJ (3)

Top Pull Request Authors

paraseba (336)
dependabot[bot] (154)
mpiannucci (106)
dcherian (106)
li-em (62)
ianhi (55)
TomNicholas (27)
rabernat (21)
jhamman (13)
jbusecke (11)
samantha-earthmover (9)
maxrjones (8)
DahnJ (6)
jakenotjay (5)
abkfenris (4)

Top Issue Labels

CI (33)
bug :bug: (22)
good first issue :hatching_chick: (17)
virtual references :ghost: (15)
documentation :memo: (15)
python :snake: (15)
enhancement :sparkles: (10)
performance :zap: (7)
bindings :link: (4)
design doc :artist: (3)
storage 💾 (3)
manifests :crystal_ball: (3)
python (3)
reprs (3)
rust :crab: (2)
testing (2)
API design :art: (2)
on disk :cd: (1)
windows (1)
breaking-change (1)
dependencies :pushpin: (1)
ic2 (1)
github actions :gear: (1)
performance (1)
packaging :gift: (1)
spec :spiral_notepad: (1)
good first issue (1)
Implementación (1)
bug (1)
help wanted :raised_hand_with_fingers_splayed: (1)

Top Pull Request Labels

dependencies (132)
rust (101)
github_actions (26)
javascript (25)
dependencies :pushpin: (22)
test-upstream (19)
documentation :memo: (8)
virtual references :ghost: (6)
python :snake: (5)
API design :art: (3)
run-integration-tests (3)
python (3)
rust :crab: (2)
design doc :artist: (2)
docs (1)
enhancement :sparkles: (1)
bug :bug: (1)
v-next (1)
publish-npm (1)
reprs (1)
breaking-change (1)

Package metadata

Total packages: 11
Total downloads:
- cargo: 46,642 total
- pypi: 248,425 last-month
- npm: 1,034 last-month
- conda: 102 total
Total dependent packages: 0 (may contain duplicates)
Total dependent repositories: 0 (may contain duplicates)
Total versions: 328
Total maintainers: 7