GridWorks SCADA

========

This code is intended for running a heat pump thermal storage space heating system in a house, and doing this transactively. That means the heating system is capable of dynamically responding to local electric grid conditions and buying energy at the lowest cost times, while keeping the house warm. We believe this repo may be instrumental in effectively and efficiently reaching a low-carbon future. For an architectural overview of the code, and why it has something to do with a low-carbon future, please go here.

This code is part of a larger framework. In particular it assumes there is a cloud-based actor which it refers to as the AtomicTNode (short for Atomic Transactive Node) that is calling the shots on its control decisions most of the time. In addition, the code is structured in an
actor-based way, with a collection of actors each responsible for an important but limited set
of functionality communicating to each other via messages. For a more specific description of both how these internal actors work with each other and how
this repo fits into the larger transactive energy framework please go here; this page describes typical sequences of messages between relevant actors in the system.

We are indebted to Efficiency Maine, who spearheaded and funded the initial pilot using this code. As per the requirements of the initial pilot, the code is intended to:

1. run on a raspberry Pi 4; and
2. to be able to use a variety of off-the-shelf actuating and sensing devices.

For information on setting up an SD card that will run this code on a Pi 4 with the correct
configuration and attached devices, please go here

Creating a Dev environment for macos or Pi

Use python 3.11

Create the development environment with:

tools/mkenv.sh

On a Pi run:

tools/mkenv-pi.sh

To activate the environment, add 'gw_spaceheat' to your PYTHONPATH and then
source the environment's activation file. This is most easily accomplished
with a shell alias, for example on a Mac by fixing to following to contain the
actual the path to this repo on your machine and then adding the result to
$HOME/.zprofile:

export SCADA_REPO=actual/path/to/your/repo
export GW_SPACEHEAT=$SCADA_REPO/gw_spaceheat
alias gw="source $GW_SPACEHEAT/venv/bin/activate && export PYTHONPATH=$GW_SPACEHEAT && cd $SCADA_REPO"

You will then be able to activate the development environment in a new terminal
with:

gw

Run the tests from the root directory of the repo with:

pytest

A hardware layout file is necessary to run the scada locally. Find the default path the layout file with:

python -c "from gwproactor.config.paths import Paths; print(Paths().hardware_layout)"

For initial experiments the test layout file can be used. The test layout file is located at:

tests/config/hardware-layout.json

Display the hardware layout with:

gws layout show

Display current settings with:

gws config

There are some scratch notes on Pi-related setup (like enabling interfaces) in docs/pi_setup.md

Adding libraries

Add libraries by adding the library spec to the appropriate ".in" file in the
requirements directory. Use:

• dev.in for requirements only needed for
  development or CI.
• drivers.in for requirements only
  needed on a Pi.
• base.in for requirements used used
  in all contexts.

Once you have added your requirement run:

tools/pipc.sh

Then manually modify the modified .txt files to remove the absolute paths
that pip-tools adds to comments inside the .txt files. This allow someone looking
at the commit to see only the dependency you changed. Otherwise they will see a
change in the comment many dependencies. For example, in a text editor or IDE
do a search/replace in *.txt files in the project, searching for the text

path/to/scada/repo/on/your/machine/gw_spaceheat

and replacing it with

gw_spaceheat

Handling secrets and config variables

SETTING UP SECRETS.
Configuration variables (secret or otherwise) use dotenv module in a gitignored .env file, copied over from .env-template. These are accessed via config.ScadaSettings.

Setting up MQTT

See instructions here to set up a local MQTT broker
using Mosquitto.

Static analsyis with ruff

Ruff is installed via the test and dev requirements.
You can run it with:

ruff check

Ruff is not run in CI or in pre-commit, since the code will not currently pass.
Ruff is provided primarily for visual feedback in the IDE. Ruff is configured
in pyproject.toml.

Static analysis in Visual Studio Code

Visual Studio Code will provide visual feedback on code that does not pass ruff.

To use this functionality, the ruff plugin
for Visual Studio code must be installed. We recommend:

1. Installing the ruff extension.
2. Disabling it.
3. Enabling for workspaces in which you want to use it, such as this one.

More static analysis

More rigid ruff rules can be applied by modifying pyproject.toml. Gwproto, for
example, uses many more rules.

Typechecking feedback can be applied in the IDE by enabling Pylance type checking
inside vscode. Change that
in user not workspace settings since much of the code will currently fail.

TLS

TLS is used by default. Follow these instructions to set up
a local self-signed Certificate Authority to create test certificates and to create certificates for the Mosquitto
broker. Note that this section
is relevant if you will connect to the Mosquitto broker from a Raspberry PI.

Create a certificate for the test ATN

gwcert key add --certs-dir $HOME/.config/gridworks/atn/certs scada_mqtt
cp $HOME/.local/share/gridworks/ca/ca.crt $HOME/.config/gridworks/atn/certs/scada_mqtt

Create a certificate for test Scada

gwcert key add --certs-dir $HOME/.config/gridworks/scada/certs gridworks_mqtt
cp $HOME/.local/share/gridworks/ca/ca.crt $HOME/.config/gridworks/scada/certs/gridworks_mqtt                    

Test generated certificates

In one terminal run:

mosquitto_sub -h localhost -p 8883 -t foo \
     --cafile $HOME/.config/gridworks/atn/certs/scada_mqtt/ca.crt \
     --cert $HOME/.config/gridworks/atn/certs/scada_mqtt/scada_mqtt.crt \
     --key $HOME/.config/gridworks/atn/certs/scada_mqtt/private/scada_mqtt.pem

In another terminal run:

mosquitto_pub -h localhost -p 8883 -t foo -m '{"bar":1}' \
     --cafile $HOME/.config/gridworks/scada/certs/gridworks_mqtt/ca.crt \
     --cert $HOME/.config/gridworks/scada/certs/gridworks_mqtt/gridworks_mqtt.crt \
     --key $HOME/.config/gridworks/scada/certs/gridworks_mqtt/private/gridworks_mqtt.pem

Verify you see {"bar":1} in the first window.

Configuring a Scada with keys that can be used with the GridWorks MQTT broker.

Use getkeys.py to
create and copy TLS to keys to a scada such that it can communicate with the actual GridWorks MQTT broker. For details
run:

python gw_spaceheat/getkeys.py --help

The overview of this process is that you need:

1. The ssh key for certbot.
2. rclone installed.
3. An rclone remote configured for your scada.
4. To construct the getkeys.py command line per its help.

Running the code

This command will show information about what scada would do if started locally:

gws run --dry-run  

This command will will start the scada locally:

gws run

These commands will start the local test ATN:

gws atn run

Development flow

Default branch is dev. Make PRs to this branch for review from your code branch. Make bug changes directly to this branch.
The first 5 homes in Millinocket are designed for beta testing. The idea here is that they run on dev, and the larger
group of houses run on main.

The main branch is protected - requires a pull request. Default pattern is PRs from dev to main.
This will also publish a new gridworks-scada-protcol package and a new grdiworks-admin package.

Packages

Motivated by the need to make gridworks-admin installable without setting up the
scada development environment, this repository contains a packages
directory for code published on PyPI as separate distribution packages. The
separate distribution packages are managed as separate by python projects, with
their own subdirectory, each containing its own pyproject.toml. Subproject
development is done using uv and publication to PyPI happens in CI after
merging to the main branch.

There are two packages:

1. gridworks-admin, the admin user interface.
2. gridworks-scada-protocol, which contains protocol messages shared by both
   Scada and admin.

See the packages directory README.md for more information.

Admin

To install admin as a tool separate from development environment:

uv tool install gridworks-admin

or

pipx install gridworks-admin

To run admin from the development environment:

gwa

To publish a new version admin to PyPI install uv, if necessary, and then:

1. Update the version field in the admin pyproject.toml, either using
   the uv version command, for example:

   cd packages/gridworks-admin
   uv version --bump patch
   

   or by manually modifying the pyproject.toml and then running uv lock.
2. Merge to main.

See the packages directory README.md for more information.

License

Distributed under the terms of the MIT license,
this repository is free and open source software.

Contributing

Contributions are very welcome.
To learn more, see the [Contributor Guide].