Satip

A library for satellite image processing providing all of the functionality necessary for retrieving, and storing EUMETSAT data.
https://github.com/openclimatefix/Satip

Category: Sustainable Development
Sub Category: Environmental Satellites

Keywords

eumetsat gis nowcasting python satellite-imagery

Keywords from Contributors

observability gan measure archiving solar battery conversation composer forecasting-models profiles

Last synced: about 12 hours ago
JSON representation

Repository metadata

Satip contains the code necessary for retrieving, transforming and storing EUMETSAT data

Host: GitHub
URL: https://github.com/openclimatefix/Satip
Owner: openclimatefix
License: mit
Created: 2020-11-02T16:49:59.000Z (over 4 years ago)
Default Branch: main
Last Pushed: 2025-04-21T17:16:07.000Z (5 days ago)
Last Synced: 2025-04-25T13:49:33.091Z (1 day ago)
Topics: eumetsat, gis, nowcasting, python, satellite-imagery
Language: Jupyter Notebook
Homepage: https://satip.readthedocs.io/
Size: 467 MB
Stars: 43
Watchers: 1
Forks: 34
Open Issues: 50
Releases: 345
Metadata Files:
- Readme: README.md
- License: LICENSE

Satip

Satip is a library for satellite image processing, and provides all of the functionality necessary for retrieving, and storing EUMETSAT data

Installation

To install the satip library please run:

pip install satip

Or if you're working in the development environment you can run the following from the directory root:

pip install -e .

Conda

Or, if you want to use conda from a cloned Satip repository:

conda env create -f environment.yml
conda activate satip
pip install -e .

If you plan to work on the development of Satip then also consider installing these development tools:

conda install pytest flake8 jedi mypy black pre-commit
pre-commit install

Development Environment

In order to contribute:

it's recommended that you use a Linux-based OS. This is currently used for all CI/CD testing, production, and development.
At the time of writing (21-Dec-23), the Python version used is 3.11 with work being done to update to Python 3.12. This is subject to updates over time.

Operation

Getting your own API key

In order to contribute to development or just test-run some scripts, you will need your own Eumetsat-API-key. Please follow these steps:

Go to https://eoportal.eumetsat.int and register an account.
You can log in and go to https://data.eumetsat.int/ to check available data services. From there go to your profile and choose the option "API key" or go to https://api.eumetsat.int/api-key/ directly.
Please make sure that you added the key and secret to your user's environment variables.

Downloading EUMETSAT Data

We have moved this to here

Converting Native files to Zarr

scripts/convert_native_to_zarr.py converts EUMETSAT .nat files to Zarr datasets, using very mild lossy JPEG-XL compression. (JPEG-XL is the "new kid on the block" of image compression algorithms). JPEG-XL makes the files about a quarter the size of the equivalent bz2 compressed files, whilst the images are visually indistinguishable. JPEG-XL cannot represent NaNs so NaNs. JPEG-XL understands float32 values in the range [0, 1]. NaNs are encoded as the value 0.025. All "real" values are in the range [0.075, 1]. We leave a gap between "NaNs" and "real values" because there is very slight "ringing" around areas of constant value (see this comment for more details). Use satip.jpeg_xl_float_with_nans.JpegXlFloatWithNaNs to decode the satellite data. This class will reconstruct the NaNs and rescale the data to the range [0, 1].

Running in Production

The live service uses app.py as the entrypoint for running the live data download for OCF's forecasting service, and has a few configuration options, configurable by command line argument or environment variable.

--api-key or API_KEY is the EUMETSAT API key

--api-secret or API_SECRET is the EUMETSAT API secret

--save-dir or SAVE_DIR is the top level directory to save the output files, a latest subfolder will be added to that directory to contain the latest data

--history or HISTORY is the amount of history timesteps to use in the latest.zarr files

--db-url or DB_URL is the URL to the database to save to when a run has finished

--use-rescaler or USE_RESCALER tells whether to rescale the satellite data to between 0 and 1 or not when saving to disk. Primarily used as backwards compatibility for the current production models, all new training and production Zarrs should use the rescaled data.

--use-iodc or USE_IODC is an option to get the IODC satellite data

Testing

To run tests, simply run pytest . from the root of the repository. To generate the test plots, run python scripts/generate_test_plots.py.

Environmental Variables

Some tests require environmental variables to be set that would be passed in by command line argument when running the code in production. These are as follows:

EUMETSAT_USER_KEY: the EUMETSAT API key
EUMETSAT_USER_SECRET: the EUMETSAT API secret

These can be added using the export command in your shell environment. To add these permanently, the export statements can be added to the configuration file for the shell environment (e.g. "~/.bashrc" if using bash).

Contributors ✨

Thanks goes to these wonderful people (emoji key):

This project follows the all-contributors specification. Contributions of any kind welcome!

Owner metadata

Name: Open Climate Fix
Login: openclimatefix
Email: [email protected]
Kind: organization
Description: Using open science to mitigate climate change
Website: openclimatefix.org
Location: London
Twitter: openclimatefix
Company:
Icon url: https://avatars.githubusercontent.com/u/48357542?v=4
Repositories: 88
Last ynced at: 2024-04-15T07:32:15.529Z
Profile URL: https://github.com/openclimatefix

GitHub Events

Total

Create event: 48
Release event: 23
Issues event: 13
Watch event: 2
Delete event: 19
Issue comment event: 44
Push event: 115
Pull request review comment event: 4
Pull request review event: 9
Pull request event: 53
Fork event: 5

Last Year

Create event: 48
Release event: 23
Issues event: 13
Watch event: 2
Delete event: 19
Issue comment event: 44
Push event: 115
Pull request review comment event: 4
Pull request review event: 9
Pull request event: 53
Fork event: 5

Committers metadata

Last synced: 4 days ago

Total Commits: 1,298
Total Committers: 24
Avg Commits per committer: 54.083
Development Distribution Score (DDS): 0.734

Commits in past year: 271
Committers in past year: 14
Avg Commits per committer in past year: 19.357
Development Distribution Score (DDS) in past year: 0.731

Name	Email	Commits
BumpVersion Action	b**n@g**s	345
Jacob Bieker	j**b@b**h	264
AyrtonB	A**n@O**m	165
peterdudfield	p**d@h**m	160
Rabscuttler	l**n@g**m	80
Jacqueline James	1****J	73
pre-commit-ci[bot]	6****]	57
Notger Heinz	n**z@g**m	34
Jack Kelly	j**k@o**g	33
allcontributors[bot]	4****]	27
devsjc	4****c	22
dependabot[bot]	4****]	11
Adimanv	a**7@g**m	6
peach280	v**8@g**m	5
Zhenbang Feng	j**n@u**u	4
TomPughe	1****e	3
Richa	4****a	2
ludobegins	l**n@g**m	1
Aryan Bhosale	3****e	1
Azah Norbline	a**e@g**m	1
Matthew Duffin	m****5	1
Simon Lemieux	1****x	1
Suleman Karigar	3****2	1
jsbaasi	7****i	1

Committer domains:

Issue and Pull Request metadata

Last synced: 1 day ago

Total issues: 139
Total pull requests: 214
Average time to close issues: 3 months
Average time to close pull requests: 24 days
Total issue authors: 17
Total pull request authors: 27
Average comments per issue: 2.92
Average comments per pull request: 1.1
Merged pull request: 160
Bot issues: 0
Bot pull requests: 64

Past year issues: 20
Past year pull requests: 66
Past year average time to close issues: about 1 month
Past year average time to close pull requests: 8 days
Past year issue authors: 6
Past year pull request authors: 11
Past year average comments per issue: 2.05
Past year average comments per pull request: 0.62
Past year merged pull request: 47
Past year bot issues: 0
Past year bot pull requests: 17

More stats: https://issues.ecosyste.ms/repositories/lookup?url=https://github.com/openclimatefix/Satip

Top Issue Authors

jacobbieker (48)
peterdudfield (43)
JackKelly (28)
Rabscuttler (4)
PaulaPerezL (2)
devsjc (2)
TomPughe (2)
phinate (1)
AyrtonB (1)
AUdaltsova (1)
gabrielelibardi (1)
tomasvanoyen (1)
LuisTellezSirocco (1)
BenediktSchulzAlitiq (1)
jsbaasi (1)

Top Pull Request Authors

peterdudfield (50)
jacobbieker (44)
dependabot[bot] (39)
allcontributors[bot] (14)
devsjc (13)
pre-commit-ci[bot] (11)
JackKelly (6)
JasonFengGit (5)
notger (4)
peach280 (4)
14Richa (3)
TomPughe (3)
Surya-29 (2)
aryanbhosale (2)
vikasgrewal16 (2)

Top Issue Labels

enhancement (57)
bug (38)
good first issue (30)
timeline (4)
help wanted (3)
documentation (2)
Meta (1)
tech-ops (1)

Top Pull Request Labels

dependencies (39)
enhancement (25)
bug (9)
documentation (3)

Package metadata

Total packages: 1
Total downloads:
- pypi: 7,825 last-month
Total dependent packages: 2
Total dependent repositories: 1
Total versions: 374
Total maintainers: 3

pypi.org: satip

Satip provides the functionality necessary for

Homepage:
Documentation: https://satip.readthedocs.io/
Licenses: MIT
Latest release: 2.12.40 (published about 1 month ago)
Last Synced: 2025-04-25T13:36:00.921Z (1 day ago)
Versions: 374
Dependent Packages: 2
Dependent Repositories: 1
Downloads: 7,825 Last month
Rankings:
- Dependent packages count: 3.13%
- Downloads: 7.325%
- Average: 10.716%
- Dependent repos count: 21.694%
Maintainers (3)

Dependencies

.github/workflows/workflows.yaml actions

actions/checkout v2 composite
actions/setup-python v2 composite
codecov/codecov-action v2 composite
iterative/setup-cml v1 composite

Dockerfile docker

ubuntu latest build

docs/requirements.txt pypi

jinja2 <3.1.0
pydoc-markdown *
readthedocs-custom-steps ==0.5.1

requirements.txt pypi

bottleneck *
click *
eumdac *
fsspec *
h5netcdf *
imagecodecs *
netCDF4 *
nowcasting_datamodel >=0.0.36
numcodecs *
numpy ==1.23.0
pandas >=1.1.4
psutil *
pyorbital *
pyresample ==1.25.0
requests *
s3fs *
satpy ==0.36.0
tqdm *
xarray ==2022.3.0
zarr *

.github/workflows/ghcr-release.yaml actions

actions/checkout v3 composite
docker/build-push-action ad44023a93711e3deb337508980b4b5e9bcdc5dc composite
docker/login-action f054a8b539a109f9f41c372932f1ae047eff08c9 composite
docker/metadata-action 98669ae865ea3cffbcbaa878cf57c20bbf1c6c38 composite

.github/workflows/release.yaml actions

setup.py pypi

environment.yml pypi

alembic ==1.13.0
blosc2 ==2.3.2
eumdac ==2.1.0
freezegun ==1.3.1
greenlet ==3.0.1
imagecodecs ==2023.9.18
iniconfig ==2.0.0
mako ==1.3.0
ndindex ==1.7
nowcasting-datamodel ==1.5.26
numpy ==1.23.2
ocf-blosc2 ==0.0.4
pluggy ==1.3.0
psycopg2-binary ==2.9.9
py-cpuinfo ==9.0.0
pydantic ==1.10.10
pytest ==7.4.3
sqlalchemy ==2.0.23
structlog ==23.2.0

Score: 16.67649881825214

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Open Sustainable Technology

Satip

Keywords

Keywords from Contributors

Repository metadata

README.md

Satip

Installation

Conda

Development Environment

Operation

Getting your own API key

Downloading EUMETSAT Data

Converting Native files to Zarr

Running in Production

Testing

Environmental Variables

Contributors ✨

Owner metadata

GitHub Events

Total

Last Year

Committers metadata

Committer domains:

Issue and Pull Request metadata

Top Issue Authors

Top Pull Request Authors

Top Issue Labels

Top Pull Request Labels

Package metadata

pypi.org: satip

Dependencies