Shyft

A cross-platform framework built around high-performancetime-series storage and analysis, providing the computational foundation for hydrological forecasting and energy-market modelling.
https://gitlab.com/shyft-os/shyft

Category: Energy Systems
Sub Category: Energy Markets

Keywords

C++ Hydrology energy-market forecasting optimization python time-series

Last synced: about 10 hours ago
JSON representation

Repository metadata

Source code for shyft. * Time-series for python and c++, including distributed storage and calculations * Hydrologic Forecasting Toolbox, high-performance flexible stacks, including calibration * Energy-market models and micro services

• Host: gitlab.com
• URL: https://gitlab.com/shyft-os/shyft
• Owner: shyft-os
• License: other
• Created: 2019-04-13T16:04:16.075Z (about 7 years ago)
• Default Branch: master
• Last Synced: 2025-11-27T21:18:41.444Z (6 months ago)
• Topics: C++, Hydrology, energy-market, forecasting, optimization, python, time-series
• Stars: 32
• Forks: 20
• Open Issues: 116
• Releases: 0

https://gitlab.com/shyft-os/shyft/blob/master/

          # Shyft – Framework for Energy-Market Modelling

High-performance framework for **time-series storage, analysis, and modelling**
of **hydrological and energy-market systems**.

[![Documentation](https://img.shields.io/badge/docs-shyft--doc-blue)](https://shyft-os.gitlab.io/shyft-doc/index.html)
[![Discussion](https://img.shields.io/badge/discussion-google--group-blue)](https://groups.google.com/forum/#!forum/shyft)
[![License](https://img.shields.io/badge/license-LGPLv3-blue.svg)](LICENSE)

---

## Overview

**Shyft** is a cross-platform framework built around **high-performance
time-series storage and analysis**, providing the computational foundation
for **hydrological forecasting and energy-market modelling**.


The project originated at **Statkraft** in collaboration with the
**University of Oslo – Department of Geosciences**, and is today used in
**operational energy-sector environments**, including **24×7 systems** within
Statkraft’s Energy Management division.


Shyft combines:

- a **high-performance C++ core**
- a **Python orchestration layer**
- **scalable compute and cache microservices**
- **enterprise-grade time-series and model storage**
- **model-driven system architecture**

The framework was created through close interdisciplinary collaboration
and continues to enable cooperation between:

- domain experts
- researchers
- software engineers
- operational IT teams

while maintaining the performance and robustness required for operational energy systems.

---

## Architecture

![Energy Market Model](./doc/image/shyft_energy_market_model.jpg)

*Figure crafted by Christine Schei Liland*

Shyft provides a modular platform composed of interoperable components
that can be deployed together or separately depending on the operational need.

These components include:

- high-performance **time-series services**
- distributed **DTSS compute services**
- **hydrological forecasting models**
- **energy-market models**
- Python-based orchestration and analysis tools

This modular structure supports **scalable microservice deployments**
and enables packaging strategies that minimize dependencies and image sizes.

---

## Key Capabilities

### Model-Centric Time-Series Platform

Shyft follows a **model-centric architecture** where domain models
(such as hydrology systems or energy-market models) are decorated with
time-series attributes and symbolic expressions.

The time-series engine provides:

- strongly typed time-series objects
- symbolic expressions
- lazy evaluation
- distributed execution
- persistent time-series storage

### Distributed Time-Series Services (DTSS)

The **Distributed Time-Series Service (DTSS)** provides scalable
server-side evaluation of time-series expressions, including:

- distributed time-series evaluation
- high-performance in-memory caching
- websocket-based APIs
- real-time subscriptions
- replication between deployments

### Scalable Algorithm Execution

Shyft separates **domain models** from the **execution of algorithms**,
allowing simulations, forecasts, and analytical workflows to scale
independently from the modelling layer.

This architecture enables **multiple algorithms to operate on the same
domain models**, allowing different simulation, forecasting, or
optimization methods to be applied **without coupling the models to a
specific algorithm implementation**.

Distributed compute services execute algorithms on shared time-series
data and domain models, with results attached back to the models for
analysis and decision support.

### Hydrological Forecasting

Shyft contains hydrological modelling tools designed for operational
forecasting environments, supporting distributed hydrological models
and uncertainty analysis.

### Energy-Market Modelling

The energy-market framework enables modelling of:

- power grid topology
- hydropower systems
- production units
- contracts and portfolios
- operational constraints

The modelling framework allows researchers, vendors, and operators to
integrate proprietary optimization and simulation algorithms.

---

## Documentation

Full documentation is available at:

https://shyft-os.gitlab.io/shyft-doc/

The documentation includes:

- system architecture
- installation guides
- Python API reference
- time-series engine documentation
- hydrology modelling
- energy-market modelling
- developer guides
- release and contribution workflow

---

## Supported Platforms

Shyft packages are automatically built and tested through the GitLab CI pipeline.

Primary development and CI testing platforms:

- **Arch Linux**
- **Fedora**
- **Windows (MSYS / MinGW)** *(legacy support for existing deployments)*

Shyft can generally be built on **any modern Linux distribution** with an up-to-date toolchain and the required dependencies (C++ compiler, CMake, and libraries).

### Distribution Repositories

Prebuilt Shyft packages are available through distribution-specific repositories:

 - Arch Linux: https://archlinux.helset.gotdns.org
 - Fedora: https://fedora.helset.gotdns.org
 - MSYS2 (ucrt64) : https://msys2.helset.gotdns.org

These repositories provide GPG-signed binary packages built from tagged releases using the documented Shyft release process.
Install and verification instructions for each distro is provided on the links above.

The repositories are currently operated as part of the project’s release infrastructure.

Refer to the documentation for the Shyft release process, run-books and installation instructions:
https://shyft-os.gitlab.io/shyft-doc/

---

## Security and Supply-Chain Transparency

Shyft emphasizes **software supply-chain security** and operational reliability.

The project uses practices including:

- verified GPG-signed commits
- signed release artifacts
- provenance metadata
- transparent dependency management
- preference for distro-maintained packages
- reproducible build pipelines

These practices support safe deployment in operational environments.

For background on secure open-source development practices, the following resources provide useful guidance:

- **Linux Foundation — Developing and Evaluating Secure Open Source Software (OSS)**  
  https://events.linuxfoundation.org/wp-content/uploads/2021/07/Developing-Secure-Open-Source-Software-OSS.pdf

- **Open Source Security Foundation (OpenSSF)**  
  https://openssf.org/

- **SLSA — Supply-chain Levels for Software Artifacts**  
  https://slsa.dev/

---

## Contributing

Shyft development follows a structured workflow involving:

- **Owners**
- **Maintainers**
- **Contributors**

Development happens on the `next-shyft-version` branch and follows a documented release process.

Contribution guidelines, development setup instructions, and workflow details are available in:

Project wiki  
https://gitlab.com/shyft-os/shyft/-/wikis

and specific https://gitlab.com/shyft-os/shyft/-/wikis/How-to/Contribute

---

## Container Images

The CI pipeline produces container images used for building and testing Shyft.

These images are available in the GitLab container registry:

https://gitlab.com/shyft-os/shyft/container_registry

---

## Authors and Project Stewardship

Shyft was initially developed by **Statkraft** in cooperation with the  
**University of Oslo – Department of Geosciences (UiO)**.

The project originated from a small, highly experienced interdisciplinary team
combining expertise in applied mathematics, hydrological science,
energy-system modelling, and software engineering.

The overall **architectural design and C++ core implementation** were primarily developed by:

- **Sigbjørn Helset** 
- **Ola Skavhaug** 

The initial **Python orchestration layer, data pipelines, and modelling integration**
were primarily developed by:

- **John F. Burkhart** 
- **Yisak Sultan Abdella** 

The hydrological modelling algorithms and their composition into operational
modelling stacks were strongly influenced by the hydrological research
experience of **John F. Burkhart** and **Yisak Sultan Abdella**, and by the
research collaboration with the **University of Oslo Department of Geosciences**.

Through this collaboration, UiO researchers have contributed important
hydrological algorithms and modelling components used within Shyft.

Notable contributions include the **SnowTiles algorithm** developed by
**Felix Matt**, and work by **Olga Silantyeva** on **TIN-based hydrological
modelling research** and the integration of the **FSM2 snow model**.
UiO researchers have also actively used Shyft in research and operational
studies, providing valuable feedback that helped shape the evolving
hydrology orchestration framework built on the high-performance C++
core and RocksDB-based services.

This complementary combination of **software architecture, operational
engineering, and hydrological science** formed the foundation of the Shyft
framework.

The project is currently maintained by a group of contributors and maintainers.


Project stewardship and long-term continuity are ensured by:

- **Sigbjørn Helset**, who acts as the **active owner** of the project and maintains the core development and build infrastructure.

Significant development, research collaboration, and operational deployment of Shyft has been supported by:

- **Statkraft AS**
- **University of Oslo – Department of Geosciences (UiO)**
- **Expert Analytics (XAL)** and its contributors



A complete list of contributors can be found in the **Contributors** section
below and in the Git history.

Copyright (C)

Sigbjørn Helset (SiH)  
John F. Burkhart (JFB)  
Ola Skavhaug (OS)  
Yisak Sultan Abdella (YAS)  
Statkraft AS

---

## Contributors

Project contributors include:

- Sigbjørn Helset 
- Ola Skavhaug 
- John Burkhart 
- Yisak Sultan Abdella 
- Felix Matt 
- Olga Silantyeva 
- Francesc Alted 
- Ludovic Pochon-Guérin 
- Eivind Lycke Melvær 
- Cecilie Granerød 
- Andreas Slyngstad 
- Bernardo de Olivera 
- Diako Darian 
- Eivind Aarnæs 
- Vinzenz Gregor Eck 
- Mindaugas Pivoras 
- Roar Emaus 
- Trygve Bærland 
- Albert O. Nybø 
- Gry M. Tveten 
- Magne Nordaas 
- Christine Schei Liland 
- Jens Askgaard 
- Ibrahim Rahmani 
- Anders Kjeldsen 
- Alexander Becker 
- Sarah Dahmen 
- Fabio Zeiser 
- Stian Angelsen 
- Joseph Bradshaw 
- Ole Andreas Grønn Ramsdal 
- John Eivind Rømma Helset 
- Benedikt Reinartz 
- Paul Christoph Bätzing-Rosenvinge 
- Eilidh Troup 
- Petar Kutlesic 
- Christian Vaugelade-Kilaas 
- Jakob Boye Løland Ørbæk 
- Jiyoung Kim 
- Lars Skodje 
- Ana Budisa 
---

## License

Shyft is released under the **LGPL v3 license**.

See the `LICENSE` file.

---

## Citation

If you use Shyft in hydrology-related research, please cite:

Burkhart, J. F., Matt, F. N., Helset, S., Sultan Abdella, Y., Skavhaug, O., and Silantyeva, O.

*Shyft v4.8: a framework for uncertainty assessment and distributed hydrologic modeling for operational hydrology*

Geosci. Model Dev., 14, 821–842, 2021  
https://doi.org/10.5194/gmd-14-821-2021
        

---

Committers metadata

Last synced: 2 days ago

Name	Email	Commits
sigbjorn	s****t@g****m	2027
John Eivind Helset	p****e@j****o	413
Paul Christoph Batzing-Rosenvinge	p****e@s****m	330
anybo	3****o@u****m	175
yisak	y****a@g****m	141
Sigbjørn Helset	s****t@s****m	138
Ola Skavhaug	o****a@x****o	117
Ludovic Pochon-Guerin	L****n@s****m	93
Francesc Alted	f****t@g****m	52
John Eivind Helset	j****t@g****m	52
Fabio Zeiser	f****r@f****o	45
Magne	m****s@g****m	43
Eivind Lyche Melvær	e****r@s****m	38
Christian Vaugelade-Kilaas	c****n@v****o	34
Ole Andreas Ramsdal	o****l@s****m	31
Bernardo de Oliveira	b****a@s****m	23
Trygve Bærland	t****d@g****m	21
felixmatt	f****t@g****o	19
petar.kutlesic	p****c@s****m	18
Andreas	p****8@p****e	17
cecilie	c****d@g****m	16
Diako Darian	d****n@g****m	16
Eirik Gjerløw	e****w@g****m	14
Jens Askgaard	j****d@g****m	13
John F. Burkhart	j****t@g****m	13
Eivind Storm Aarnæs	e****d@x****o	11
Felix Matt	f****t@s****m	11
Eivind Lyche Melvær	e****d@e****o	10
Eivind Storm Aarnæs	m****l@e****m	10
Jakob Boye Løland Ørbæk	j****k@s****m	10
and 43 more...

Committer domains:

• statkraft.com: 20
• xal.no: 4
• users.noreply.gitlab.com: 2
• geo.uio.no: 2
• jehelset.no: 1
• fys.uio.no: 1
• vaugelade.no: 1
• pm.me: 1
• elm.no: 1
• eistaa.com: 1
• frisurf.no: 1
• energycorp.com: 1
• eck-mail.eu: 1
• emaus.no: 1
• oslwvsht001p.energycorp.com: 1
• stud.ntnu.no: 1
• a3.home: 1
• nve.no: 1
• oslxvbld001t.energycorp.com: 1
• continuum.io: 1

---

Issue and Pull Request metadata

Last synced: 4 days ago

---

Package metadata

• Total packages: 2
• Total downloads:
  • pypi: 409 last-month
• Total dependent packages: 0 (may contain duplicates)
• Total dependent repositories: 1 (may contain duplicates)
• Total versions: 437
• Total maintainers: 2

---

Dependencies

Score: 15.308695064262032