Open Energy Benchmark

This repository contains code for benchmarking optimization solvers on problems from the energy planning domain, and an interactive website for analyzing the results. The live website can be viewed at:

https://openenergybenchmark.org/

Table of Contents

• Benchmark Problems
• Solvers
• Project Structure
• Running Benchmarks
  • Local Runs
    • Prerequisites
      • System Requirements
      • Required Software
    • Running Supported Solvers on Benchmarks
  • Running with Docker
  • Cloud Runs
• Running the Website
• Development

Benchmark Problems

All our benchmark problems are open and available as LP/MPS files that can be downloaded in one click from our website's Benchmark Problem Set page. Some problems have been generated by us using open source energy modelling frameworks, and for these we have configuration files and instructions for reproducing the problems.

For more details on how to contribute benchmark problems, see the Benchmarks README.

Solvers

The benchmark runner can run solvers listed in the Solvers README.
We use the last released solver version in each calendar year. 2025 solvers will be updated at the end of the year.

Project Structure

Understanding the project layout to help you navigate and contribute:

solver-benchmark/
├── runner/                     # Benchmark execution scripts
│   ├── benchmark_all.sh        # Main entry point for running benchmarks
│   ├── run_benchmarks.py       # Python script that orchestrates benchmark runs
│   ├── run_solver.py           # Individual solver runner
│   ├── envs/                   # Conda environment definitions for each solver year
│   └── benchmarks/             # Downloaded benchmark problem files
├── benchmarks/                 # Benchmark problem definitions and metadata
│   ├── pypsa/                  # PyPSA-generated energy models
│   ├── jump_highs_platform/    # JuMP/HiGHS benchmark metadata
│   └── *_metadata.yaml         # Problem definitions and details
├── website/             # Next.js website for viewing results
├── infrastructure/             # GCP VM deployment scripts (for running benchmarks at scale)
├── results/                    # Output directory for benchmark results
    ├── benchmark_results.csv   # Main results file
    └── metadata.yaml           # Merged metadata of all problems on the website

Running Benchmarks

Local Runs

Prerequisites

System Requirements

The benchmark runner currently requires Linux as it uses systemd-run to enforce memory limits on solvers, which is not available on macOS or Windows.

Supported Linux distributions:

• Ubuntu 20.04 LTS or later
• Debian 11 or later
• Other systemd-based Linux distributions

macOS / Windows users: Use the provided Docker image to run benchmarks in a Linux container. See Running with Docker below.

Required Software

Ensure you have the following installed:

• Python 3.12+
• Conda (install Miniconda)
• systemd (usually pre-installed on modern Linux distributions)

Running Supported Solvers on Benchmarks

The benchmark runner script (runner/benchmark_all.sh) is the main entry point for running benchmarks. It takes a list of solvers and a list of years as arguments, and runs the benchmarks for each solver and year. It creates conda environments containing the solvers and other necessary prerequisites, so a virtual environment is not necessary just for running the benchmark runner. See README .

Quickstart:

1. Run benchmarks

./runner/benchmark_all.sh -s "highs scip" -y "2025" infrastructure/benchmarks/sample_run/standard-00.yaml

2. View logs and results

tail results/benchmark_results.csv # will overwrite currently committed results
tail runner/logs/*

3. View and analyze results by running the website locally

The script will save the measured runtime and memory consumption into a CSV file in results/ that the website will then read and display. Running the website locally will allow you to view and analyze results in a user friendly way. It will use the results from results/benchmark_results.csv.

runner/benchmark_all.sh uses runner/run_benchmarks.py to run the benchmarks by year. If you wish to run benchmarks directly, you can set up the requisite conda env manually. See Documentation.

Running with Docker

You can use Docker to run benchmarks in a Linux container that has all the required dependencies.

# Build the runner image
docker build -t solver-benchmark-runner -f runner/Dockerfile .

# Run all solvers across all years, results are written to host via a volume mount
docker run --rm \
  -v $(pwd)/results:/solver-benchmark/results \
  solver-benchmark-runner results/metadata.yaml

The container accepts the same flags as benchmark_all.sh (e.g. -s, -y). Memory limit enforcement via systemd-run is not available inside Docker and is skipped automatically. For more details on available options, conda env caching, and Gurobi licensing, see the runner Docker documentation.

Cloud Runs

We have cloud orchestration setup for running benchmarks on Google Cloud Platform. See Documentation.

Quickstart:

For cloud infrastructure setup, install:

• OpenTofu
• Google Cloud SDK

gcloud auth application-default login
cd infrastructure
tofu init
tofu apply -var-file benchmarks/sample_run/run.tfvars

To set up comprehensive benchmark campaigns, like the one available on the website:

1. Use notebooks/allocate-benchmarks-to-vms.ipynb to create the benchmark campaign.
2. Run notebooks/run-and-observe-benchmarks.ipynb to observe the benchmark campaign progress.

Alternatively, the benchmark campaign generation can be performed from the command line using the tools described below.

Benchmark campaign creation

This repository contains tooling to create benchmark campaigns from the benchmark metadata.

Campaigns can target either:

• cloud execution using Google Cloud Platform VMs (currently allowed to maintainers only);
• local execution using the existing benchmark runner workflow.

The main entry point is:

python benchmarks/create_benchmark_campaign.py

The script prepares benchmark metadata, selects benchmark instances, and creates
either:

• a cloud campaign under infrastructure/benchmarks/<run-id>/; or
• a local campaign under infrastructure/local/benchmarks/<run-id>/.

Execution targets

The campaign generator supports two execution targets:

The default target is:

cloud

Override it with:

python benchmarks/create_benchmark_campaign.py \
  --target local

to launch a benchmark campaign on your local machine.

Basic usage

Create a campaign for all benchmark instances (please do this very carefully, especially if running on the cloud):

python benchmarks/create_benchmark_campaign.py \
  --campaign my-test \
  --all

This creates a run ID of the form:

YYYYMMDD-my-test

Configuration files

Campaigns can also be defined through a YAML configuration file.

A complete template is provided in:

benchmarks/config.campaign.default.yaml

Run a campaign directly from a configuration file:

python benchmarks/create_benchmark_campaign.py \
  --configfile benchmarks/config.campaign.default.yaml

Command-line arguments always override values defined in the configuration file.

For example:

python benchmarks/create_benchmark_campaign.py \
  --configfile benchmarks/config.campaign.default.yaml \
  --campaign my-test \
  --timeout-hours 6

Select benchmarks

Select all instances of one benchmark:

python benchmarks/create_benchmark_campaign.py \
  --campaign pypsa-eur-test \
  --benchmark pypsa-eur

Select benchmark instances by size class:

python benchmarks/create_benchmark_campaign.py \
  --campaign pypsa-eur-small-medium \
  --benchmark pypsa-eur-elec \
  --size S M

Select benchmark instances by metadata name, for example specific spatial or temporal resolutions (field Name in the metadata):

python benchmarks/create_benchmark_campaign.py \
  --campaign pypsa-eur-resolution-test \
  --benchmark pypsa-eur-elec \
  --name 2-1h 3-2h

Combine size and name filters:

python benchmarks/create_benchmark_campaign.py \
  --campaign pypsa-eur-filtered \
  --benchmark pypsa-eur-elec \
  --size S M \
  --name 2-1h 3-2h

Select mixed benchmark instances explicitly:

python benchmarks/create_benchmark_campaign.py \
  --campaign mixed-test \
  --instance pypsa-eur:2-1h \
  --instance pypsa-earth:3-2h

The instance selector uses the format:

<benchmark-name>:<instance-name>

For example:

PyPSA-DE:10-1h

Include metadata entries marked as skipped (due to known timeout or memory issues):

python benchmarks/create_benchmark_campaign.py \
  --campaign clean-test \
  --all \
  --do-not-skip

Cloud VM allocation

This section only applies to cloud campaigns.

By default, the script creates one VM per selected benchmark instance.

Use a custom number of VMs with:

python benchmarks/create_benchmark_campaign.py \
  --campaign packed-test \
  --all \
  --num-vms 5

Cloud machine settings

By default, benchmark instances are assigned to VM profiles automatically based on their metadata size class:

Default zone:

us-central1-a

When --machine-type is specified, all selected benchmark instances use the chosen profile regardless of their size classification.

Timeout policy

If no timeout is provided, the script applies the default timeout policy:

S/M instances: 1 hour
L instances:   24 hours

Override the timeout:

python benchmarks/create_benchmark_campaign.py \
  --campaign timeout-test \
  --all \
  --timeout-hours 6

Solver and year selection

By default, the generated VM YAML files benchmark all the following solvers:

gurobi highs scip cbc glpk

The default solver year is:

2025

The year corresponds to the benchmark conda environment:

benchmark-<year>

For example:

2025 -> benchmark-2025

Run specific solvers for one or more years:

python benchmarks/create_benchmark_campaign.py \
  --campaign year-test \
  --all \
  --solver cbc highs \
  --years 2024 2025

Campaign summary

Every generated campaign includes a:

campaign_summary.csv

containing the benchmark selection, campaign configuration, solver selection,
timeout settings, allocation decisions, and metadata used to create the campaign.

Existing campaign directories

The script fails if the target campaign directory already exists.

Use a different campaign name, remove the existing directory, or overwrite it:

python benchmarks/create_benchmark_campaign.py \
  --campaign my-test \
  --all \
  --force

Launching a cloud campaign

After reviewing the generated files, launch the campaign from the infrastructure directory:

cd infrastructure

tofu apply \
  -var-file benchmarks/<run-id>/run.tfvars \
  -state=states/<run-id>.tfstate

Launching a local campaign

Generate a local campaign:

python benchmarks/create_benchmark_campaign.py \
  --target local \
  --campaign my-local-run \
  --benchmark pypsa-de-elec

This creates:

infrastructure/local/benchmarks/<run-id>/
├── campaign_summary.csv
├── local_benchmarks.yaml
└── run_local.sh

By default, local campaigns ask for confirmation before execution.

The generated run script can also be executed manually:

bash infrastructure/local/benchmarks/<run-id>/run_local.sh

Local campaigns use the existing runner/benchmark_all.sh workflow and execute benchmark instances sequentially.

Example workflows

Cloud campaign

python benchmarks/create_benchmark_campaign.py \
  --configfile benchmarks/config.campaign.default.yaml \
  --campaign pypsa-de-scaling

cd infrastructure

tofu apply \
  -var-file benchmarks/<run-id>/run.tfvars \
  -state=states/<run-id>.tfstate

Local campaign

python benchmarks/create_benchmark_campaign.py \
  --configfile benchmarks/config.campaign.default.yaml \
  --target local \
  --campaign pypsa-de-local

The campaign generator creates the local benchmark files and asks whether the benchmark run should be started immediately.

Running your own benchmarks

To run your own benchmark problems, either locally or on the cloud, follow the steps in the appropriate section above but using a benchmarks.yaml file of your own that gives the details (metadata) and URL/path of your benchmark problems.
Here is a small example:

benchmarks:
  genx-3_three_zones_w_co2_capture-no_uc:
    Sizes:
    - Name: 3-1h
      # Size classification
      Size: M
      # URL of the problem (needed for cloud runs)
      URL: https://storage.googleapis.com/solver-benchmarks/genx-3_three_zones_w_co2_capture-no_uc.lp
      # ALTERNATIVELY, for local runs, you can also give a local path
      Path: tests/sample_benchmarks/sample_lp.lp

You can quickly try running your own problem locally on our supported set of solvers by following these instructions.

Running other solvers

To run either our benchmarks, or your own (see the previous section), on a solver that we do not yet support, you need to install it into the active conda evironment and modify the run_solver.py appropriately. Please reach out to us (or open an issue) if you would like more details, or any help with this.

Running the Website

The website code is under website/. To run the website locally, you need a recent version of node and npm installed. Then, run the following commands:

cd website/
npm install
npm run build && npm run dev

Open http://localhost:3000 with your browser to see the website.

To see the results from your runs, navigate to the results page.

Development

We use the ruff code linter and formatter, and GitHub Actions runs various pre-commit checks to ensure code and files are clean.

You can install a git pre-commit that will ensure that your changes are formatted
and no lint issues are detected before creating new commits:

pip install pre-commit
pre-commit install

If you want to skip these pre-commit steps for a particular commit, you can run:

git commit --no-verify