Getting Started with TEMOA and Version 3

Overview

The main subdirectories in the project are:

1. temoa/
   Contains the core Temoa model

2. temoa/temoa_model
   The core model code necessary to build and solve a Temoa instance

3. temoa/data_processing
   Code for post-processing solved models and working with output

4. temoa/extensions
   Model extensions to solve the model using differing techniques. Note: There is some legacy and non-working
   code in these modules that is planned future work.

5. data_files/
   Intended to hold input data files and config files. Examples are included.
   Note that the example file utopia.sql represents a simple system called 'Utopia', which
   is packaged with the MARKAL model generator and has been used
   extensively for benchmarking exercises.

6. output_files/
   The target for run-generated output including log files and other requested products. Temoa will create
   time-stamped folders to gather output for runs

7. docs/
   Contains the source code for the Temoa project manual, in reStructuredText
   (ReST) format.

8. notebooks/
   jupyter notebooks associated with the project. Note: Not all of these are functional at this time, but are
   retained to guide future development

Guide to Setup

1. Obtain a current copy of Python from the python.org website. The model has been tested with 3.11 and 3.12. It will
   fail (raise error) on earlier versions.
2. A requirements.txt file has been included to allow for use of pip to populate a virtual environment. In order to use that the steps are:

• Ensure you have a copy of python 3.11/3.12 installed on your machine (python.org)
• Make and activate a virtual environment using the venv package:

$ python3.11 -m venv venv
$ source venv/bin/activate   # for linux/osx, windows activation command may differ

• Verify that you have a prepended indicator on your cursor that you are in the virtual environment (see below)
• After activating the venv, use pip within the venv to install everything. Most IDEs have automated tools to
  help set up and associate this venv with the project. It is also possible from the command line:

(venv) $ pip install -r requirements.txt

• For Conda users, an environment.yml file is provided that is not currently fully tested. Additional installs may
  be required.

3. The entry point for regular execution is now at the top level of the project so a "sample" run should be initiated as:

(venv) temoa $ python main.py --config data_files/my_configs/config_sample.toml

Database Setup

• Several sample database files in Version 3 format are provided in SQL format for learning/testing. These are provided in the
  data_files/example_dbs folder. In order to use them, they must be converted into sqlite database files. This can
  be done from the command line using the sqlite3 engine to convert them. sqlite3 is packaged with Python and should be
  available. If not, most configuration managers should be able to install it. The command to make the .sqlite file
  is (for Utopia as an example):

(venv) $ sqlite3 utopia.sqlite < utopia.sql

• Converting legacy db's to Version 3 can be done with the included database migration tool. Users who use this
  tool are advised to carefully review the console outputs during conversion to ensure accuracy and check the
  converted database carefully. The migration tool will build an empty new Version 3 database and move data from
  the old database, preserving the legacy database in place. The command can be run from the top level of the
  project and needs pointers to the target database and the Version 3 schema file. A typical execution from top level
  should look like:

(venv) $ python temoa/utilities/db_migration_to_v3.py --source data_files/<legacy db>.sqlite  --schema data_files/temoa_schema_v3.sql

• Users may also create a blank full or minimal version of the database from the two schema files in the data_files
  directory as described above using the sqlite3 command. The "minimal" version excludes some of the group
  parameters and is recommended as a starting point for entry-level models. It can be upgraded to the full set of
  tables by executing the full schema SQL command on the resulting database later, which will add the missing tables.

Config Files

• A configuration (config) file is required to run the model. The sample_config.toml is provided as a reference
  and has all parameters in it. It can be copied/renamed, etc.
• Notes on Config Options:

Currently Supported Modes

Check

Build the model and run the numerous checks on it. Results will be in the log file. No solve is attempted.
Note: The LP file for the model can be saved with this option and solved later/independently by selecting
the save_lp_file option in the config.

Perfect Foresight

All-in-one run that solves the entire model at once. It is possible to run this without source tracing, which will
use raw data in the model without checking the integrity of the underlying network. It is highly advised to use
source tracing for most accurate results.

Myopic

Solve the model sequentially through iterative solves based on Myopic settings. Source tracing is required to
accommodate build/no-build decisions made per iteration to ensure follow-on models are well built.

MGA (Modeling to Generate Alternatives)

An iterative solving process to explore near cost-optimal solutions. See the documentation on this mode.

SVMGA (Single Vector MGA)

A sequence of 2 model solves that establishes a base optimal cost, then relaxes the cost then minimizes an
alternate unweighted objective function comprised of variables associated with labels selected in lists in the
config file.

Method of Morris

A limited sensitivity analysis of user-selected variables using a Method of Morris approach. See the documentation
on this mode.

Build Only

Mostly for test/troubleshooting. This builds/returns an un-solved model.

Several other options are possible to pass to the main execution command including changing the logging level to
debug or running silent (no console feedback) which may be best for server runs. Also, redirecting the output
products is possible. To see available options invoke the main.py file with the -h flag:

(venv) $ python main.py -h

Typical Run

1. Prepare a database (or copy of one) as described above. Runs will fill the output tables and overwrite any data with the
   same scenario name.
2. Perepare a config file with paths to the database(s) relative to the top of the project, as in the example
3. Run the model, using the main.py entry point from the top-level of the project:

(venv) temoa $ python main.py --config data_files/my_configs/config_sample.toml

4. Review the config display and accept
5. Review the log file and output products which are automatically placed in a time-stamped folder in output_files,
   unless user has redirected output
6. Review the data in the Output tables

Testing

Users who wish to exercise the pytest based test in the test folder can do so from the command line or any IDE.
Note that many of the tests perform solves on small models using the freely available cbc solver, which is
required to run the testing suite.

The tests should all run and pass (several are currently skipped and reflect in-process work). Tests should normally
be run from the top level of the tests folder. If pytest is installed it will locate tests within the folder and
run/report them. Note the dot '.' below indicating current folder:

(venv) temoa/tests pytest .

Several of the packages used may currently generate warnings during this testing process, but the tests should all PASS
with the exception of skipped tests.

Documentation and Additional Information

The full Temoa documentation can be built by following the build README file in the Documentation folder.

Hot Fix for Network Plots on Windows Machines

Users wishing to utilize the feature to make the html network plots of the energy network using the
plot_commodity_network option in the config file who are working on Windows Operating System may need to make a
"hot fix" to the library code. See note here: https://github.com/robert-haas/gravis/issues/10

The gravis library which nicely makes these plots appears to currently be non-maintained and a 1-line fix is
likely needed to avoid error on Windows machines:

1. Within the venv that contains project dependencies, navigate to the gravis folder

2. Open the file gravis/_internal/plotting/data_structures.py and edit line 120 to include the encoding flag:

   with open(filepath, 'w', encoding='utf-8') as file_handle:

Hot Fix for Graphviz

Users wishing to utilize the graphviz package to visualize results as described in the README.md file
in the data_processing package/folder may need to re-install graphviz using another delivery means
other than pip. The current requirements.txt will attempt to install graphviz, but according to
their project page, this needs to be done with another configuration manager like apt or homebrew.

Mac users wishing to use graphviz should re-install using homebrew with the command:

brew install graphviz

(Any Windows users who have tips/info on this are asked to submit a PR to this file to update this section.)