Data

Tools

Indexes

Applications

Experiments

Open Sustainable Technology

Scores

A Python package of mathematical functions for the verification, evaluation and optimisation of forecasts, predictions or models, primarily supporting the meteorological, climatological and geoscientific communities.
https://github.com/nci/scores

Category: Climate Change
Sub Category: Earth and Climate Modeling

Keywords

climate contingency-table dask forecast-evaluation forecast-verification forecasting model-validation oceanography pandas python verification weather xarray

Keywords from Contributors

optimize archiving transforms measur generic compose animals observation projection conversion

Last synced: about 17 hours ago
JSON representation

Repository metadata

scores: Metrics for the verification, evaluation and optimisation of forecasts, predictions or models.

README.md

scores: Verification and Evaluation for Forecasts and Models

DOI CodeQL Coverage Status Binder PyPI Version Conda Version

A list of over 60 metrics, statistical techniques and data processing tools contained in scores is available here.

scores is a Python package containing mathematical functions for the verification, evaluation and optimisation of forecasts, predictions or models. It supports labelled n-dimensional (multidimensional) data, which is used in many scientific fields and in machine learning. At present, scores primarily supports the geoscience communities; in particular, the meteorological, climatological and oceanographic communities.

Documentation: scores.readthedocs.io
Source code: github.com/nci/scores
Tutorial gallery: available here
Journal paper: scores: A Python package for verifying and evaluating models and predictions with xarray

If you use scores for your work or a publication, please cite
our paper.

Overview

Below is a curated selection of the metrics, tools and statistical tests included in scores. (Click here for the full list.)

Description Selection of Included Functions
Continuous Scores for evaluating single-valued continuous forecasts. E.g. MAE, MSE, RMSE, Bias, Pearson's Correlation Coefficient, Kling-Gupta Efficiency, NSE, Flip-Flop Index, Quantile Loss, Quantile Interval Score, Interval Score, and threshold weighted scores for expectiles, quantiles and Huber Loss. See all.
Probability Scores for evaluating forecasts that are expressed as predictive distributions, ensembles, and probabilities of binary events. E.g. Brier Score, CRPS for CDFs and ensembles (including threshold weighted versions), and Isotonic Regression (reliability diagrams). See all.
Categorical Scores for evaluating forecasts of categories. E.g. 18 binary contingency table (confusion matrix) metrics, the FIxed Risk Multicategorical (FIRM) Score, the SEEPS score and the Risk Matrix Score. See all.
Spatial Scores that take into account spatial structure. Fractions Skill Score. See all.
Statistical Tests Tools to conduct statistical tests and generate confidence intervals. Diebold Mariano. See all.
Processing Tools Tools to pre-process data. E.g. Data matching, Discretisation, Block Bootstrapping, and Cumulative Density Function Manipulation. See all.
Plotting Data Tools to generate data for plotting. ROC curves, Murphy diagrams, and Q-Q plots. See all.
Emerging Emerging scores that are still undergoing mathematical peer review. They may change in line with the peer review process. Note - the Risk Matrix Score has recently been moved to 'categorical' following peer-reviewed publication.

scores not only includes common scores (e.g., MAE, RMSE), it also includes novel scores not commonly found elsewhere (e.g., FIRM, Flip-Flop Index), complex scores (e.g., threshold weighted CRPS), and statistical tests (e.g., the Diebold Mariano test). Additionally, it provides pre-processing tools for preparing data for scores in a variety of formats including cumulative distribution functions (CDF). scores provides its own implementations where relevant to avoid extensive dependencies.

scores primarily supports xarray datatypes for Earth system data allowing it to work with NetCDF4, HDF5, Zarr and GRIB data formats among others. scores uses Dask for scaling and performance. Some metrics work with pandas and we aim to expand this capability.

All of the scores and metrics in this package have undergone a thorough scientific and software review. Every score has a companion Jupyter Notebook tutorial that demonstrates its use in practice.

Contributing

Contributions from the community are warmly welcomed. To find out more, see our contributing guide.

All interactions in discussions, issues, emails and code (e.g., pull requests, code comments) will be managed according to the expectations outlined in the code of conduct and in accordance with all relevant laws and obligations. This project is an inclusive, respectful and open project with high standards for respectful behaviour and language. The code of conduct is the Contributor Covenant, adopted by over 40,000 open source projects. Any concerns will be dealt with fairly and respectfully, with the processes described in the code of conduct.

Installation

The installation guide describes four different use cases for installing, using and working with this package.

Most users currently want the all installation option. This includes the mathematical functions (scores, metrics, statistical tests etc.), the tutorial dependencies and development libraries.

# From a local checkout of the Git repository
pip install -e ".[all]"

To install the mathematical functions ONLY (no tutorial dependencies, no developer libraries), use the default minimal installation option. minimal is a stable version with limited dependencies. This can be installed from the Python Package Index (PyPI) or with conda.

# From PyPI
pip install scores

# From conda-forge
conda install conda-forge::scores

(Note: at present, only the minimal installation option is available from conda. In time, we intend to add more installation options to conda.)

Using scores

Here is a short example of the use of scores:

> import scores
> forecast = scores.sample_data.simple_forecast()
> observed = scores.sample_data.simple_observations()
> mean_absolute_error = scores.continuous.mae(forecast, observed)
> print(mean_absolute_error)
<xarray.DataArray ()>
array(2.)

Jupyter Notebook tutorials are provided for each metric and statistical test in scores, as well as for some of the key features of scores (e.g., dimension handling and weighting results).

To watch a PyCon AU 2024 conference presentation about scores click here.

Finding, Downloading and Working With Data

All metrics, statistical techniques and data processing tools in scores work with xarray. Some metrics work with pandas. As such, scores works with any data source for which xarray or pandas can be used. See the data sources page and this tutorial for more information on finding, downloading and working with different sources of data.

Archives of scores on Zenodo

scores is archived on Zenodo. Click here to see the latest version on Zenodo.

Acknowledging or Citing scores

If you use scores for a published work, we would appreciate you citing our paper:

Leeuwenburg, T., Loveday, N., Ebert, E. E., Cook, H., Khanarmuei, M., Taggart, R. J., Ramanathan, N., Carroll, M., Chong, S., Griffiths, A., & Sharples, J. (2024). scores: A Python package for verifying and evaluating models and predictions with xarray. Journal of Open Source Software, 9(99), 6889. https://doi.org/10.21105/joss.06889

BibTeX:

@article{Leeuwenburg_scores_A_Python_2024,
author = {Leeuwenburg, Tennessee and Loveday, Nicholas and Ebert, Elizabeth E. and Cook, Harrison and Khanarmuei, Mohammadreza and Taggart, Robert J. and Ramanathan, Nikeeth and Carroll, Maree and Chong, Stephanie and Griffiths, Aidan and Sharples, John},
doi = {10.21105/joss.06889},
journal = {Journal of Open Source Software},
month = jul,
number = {99},
pages = {6889},
title = {{scores: A Python package for verifying and evaluating models and predictions with xarray}},
url = {https://joss.theoj.org/papers/10.21105/joss.06889},
volume = {9},
year = {2024}
}

Citation (CITATION.cff)

cff-version: "1.2.0"
authors:
- family-names: Leeuwenburg
  given-names: Tennessee
  orcid: "https://orcid.org/0009-0008-2024-1967"
- family-names: Loveday
  given-names: Nicholas
  orcid: "https://orcid.org/0009-0000-5796-7069"
- family-names: Ebert
  given-names: Elizabeth E.
- family-names: Cook
  given-names: Harrison
  orcid: "https://orcid.org/0009-0009-3207-4876"
- family-names: Khanarmuei
  given-names: Mohammadreza
  orcid: "https://orcid.org/0000-0002-5017-9622"
- family-names: Taggart
  given-names: Robert J.
  orcid: "https://orcid.org/0000-0002-0067-5687"
- family-names: Ramanathan
  given-names: Nikeeth
  orcid: "https://orcid.org/0009-0002-7406-7438"
- family-names: Carroll
  given-names: Maree
  orcid: "https://orcid.org/0009-0008-6830-8251"
- family-names: Chong
  given-names: Stephanie
  orcid: "https://orcid.org/0009-0007-0796-4127"
- family-names: Griffiths
  given-names: Aidan
- family-names: Sharples
  given-names: John
message: If you use this software, please cite our article in the
  Journal of Open Source Software.
title: "scores: A Python package for verifying and evaluating models and
  predictions with xarray"  
preferred-citation:
  authors:
  - family-names: Leeuwenburg
    given-names: Tennessee
    orcid: "https://orcid.org/0009-0008-2024-1967"
  - family-names: Loveday
    given-names: Nicholas
    orcid: "https://orcid.org/0009-0000-5796-7069"
  - family-names: Ebert
    given-names: Elizabeth E.
  - family-names: Cook
    given-names: Harrison
    orcid: "https://orcid.org/0009-0009-3207-4876"
  - family-names: Khanarmuei
    given-names: Mohammadreza
    orcid: "https://orcid.org/0000-0002-5017-9622"
  - family-names: Taggart
    given-names: Robert J.
    orcid: "https://orcid.org/0000-0002-0067-5687"
  - family-names: Ramanathan
    given-names: Nikeeth
    orcid: "https://orcid.org/0009-0002-7406-7438"
  - family-names: Carroll
    given-names: Maree
    orcid: "https://orcid.org/0009-0008-6830-8251"
  - family-names: Chong
    given-names: Stephanie
    orcid: "https://orcid.org/0009-0007-0796-4127"
  - family-names: Griffiths
    given-names: Aidan
  - family-names: Sharples
    given-names: John
  date-published: 2024-07-09
  doi: 10.21105/joss.06889
  issn: 2475-9066
  issue: 99
  journal: Journal of Open Source Software
  publisher:
    name: Open Journals
  start: 6889
  title: "scores: A Python package for verifying and evaluating models
    and predictions with xarray"
  type: article
  url: "https://joss.theoj.org/papers/10.21105/joss.06889"
  volume: 9

Owner metadata

GitHub Events

Total
Last Year

Committers metadata

Last synced: about 2 months ago

Total Commits: 744
Total Committers: 25
Avg Commits per committer: 29.76
Development Distribution Score (DDS): 0.512

Commits in past year: 232
Committers in past year: 16
Avg Commits per committer in past year: 14.5
Development Distribution Score (DDS) in past year: 0.638

Name Email Commits
Tennessee Leeuwenburg t****g@b****u 363
Stephanie Chong 1****g 151
Nicholas Loveday 4****y 104
Nikeeth Ramanathan n****n@g****m 17
Aidan Griffiths a****s@b****u 13
Harrison Cook h****k@b****u 11
reza-armuei 1****i 9
Arshia Sharma a****a@a****u 8
dependabot[bot] 4****] 8
Deryn 1****s 8
John Sharples j****s@b****u 7
Maree Carroll m****l@g****m 6
rob-taggart 8****t 6
Aidan Griffiths 5****s 5
Liam Bluett 8****t 5
arshia 6****r 4
Beth Ebert b****t@b****u 3
Paul Smith p****u@g****m 3
durgals d****a@g****m 2
Samuel Bishop l****s@m****m 2
Justin Smallwood j****d@g****m 2
JinghanFu 1****u 2
Dougie Squire 4****e 2
AJTheDataGuy 1****y 2
thomaspagano t****o@b****u 1

Committer domains:

Issue and Pull Request metadata

Last synced: about 2 months ago

Total issues: 376
Total pull requests: 694
Average time to close issues: about 1 month
Average time to close pull requests: 7 days
Total issue authors: 24
Total pull request authors: 25
Average comments per issue: 1.28
Average comments per pull request: 1.83
Merged pull request: 557
Bot issues: 0
Bot pull requests: 11

Past year issues: 87
Past year pull requests: 185
Past year average time to close issues: 14 days
Past year average time to close pull requests: 7 days
Past year issue authors: 14
Past year pull request authors: 15
Past year average comments per issue: 1.31
Past year average comments per pull request: 2.86
Past year merged pull request: 151
Past year bot issues: 0
Past year bot pull requests: 0

More stats: https://issues.ecosyste.ms/repositories/lookup?url=https://github.com/nci/scores

Top Issue Authors

  • tennlee (133)
  • nicholasloveday (82)
  • Steph-Chong (75)
  • nikeethr (17)
  • rob-taggart (13)
  • mareecarroll (12)
  • aidanjgriffiths (8)
  • savente93 (7)
  • calebweinreb (5)
  • HCookie (5)
  • durgals (4)
  • John-Sharples (2)
  • lluritu (2)
  • rbeucher (1)
  • jdgsmallwood (1)

Top Pull Request Authors

  • tennlee (227)
  • Steph-Chong (225)
  • nicholasloveday (94)
  • nikeethr (19)
  • HCookie (19)
  • reza-armuei (14)
  • dependabot[bot] (11)
  • aidanjgriffiths (9)
  • mareecarroll (9)
  • rob-taggart (9)
  • arshiaar (8)
  • durgals (8)
  • JinghanFu (6)
  • DerynGriffiths (6)
  • lbluett (6)

Top Issue Labels

  • good first issue (19)
  • documentation (17)
  • enhancement (10)
  • refactoring (6)
  • advanced (4)
  • new metric (3)
  • intermediate (3)
  • investigation (2)
  • bug (2)
  • question (2)
  • coding (1)

Top Pull Request Labels

  • dependencies (11)
  • enhancement (5)
  • documentation (4)
  • emerging (3)
  • wontfix (1)
  • experimental (1)
  • good first issue (1)

Package metadata

pypi.org: scores

Scores is a Python package containing mathematical functions for the verification, evaluation and optimisation of forecasts, predictions or models.

  • Homepage:
  • Documentation: https://scores.readthedocs.io/en/stable/
  • Licenses: Apache Software License
  • Latest release: 2.3.0 (published 2 months ago)
  • Last Synced: 2025-10-29T20:09:45.541Z (about 2 months ago)
  • Versions: 27
  • Dependent Packages: 0
  • Dependent Repositories: 2
  • Downloads: 2,028 Last month
  • Rankings:
    • Dependent packages count: 9.973%
    • Average: 11.102%
    • Dependent repos count: 11.632%
    • Downloads: 11.703%
  • Maintainers (1)

Dependencies

environment.yml conda
  • pip
.github/workflows/python-app.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v3 composite
docs/requirements.txt pypi
  • bottleneck *
  • myst-parser *
  • pandas *
  • scipy *
  • scores *
  • sphinx *
  • sphinx-book-theme *
  • xarray *

Score: 16.426146366959443