Data

Tools

Indexes

Applications

Experiments

Open Sustainable Technology

Recent Releases of Calliope

Calliope - Release v0.7.0.dev8

User-facing changes

|changed| Rename definition_matrix input array to active, indexed over only nodes and techs (no longer also carriers), to align with YAML active definition.

|changed| |backwards-incompatible| foreach now restricts components to active node/tech combinations only; it no longer implicitly restricts the carriers dimension. Carrier membership must be stated explicitly in a component's where string (e.g., carrier_out, or a reference to a decision variable that is itself gated on carriers).

|removed| |backwards-incompatible| defined math helper function no longer available. Users should now define a boolean lookup array in their inputs to access the same functionality.

|new| init config key retain_inactive which allows deactivated components to remain in the input array rather than be cleaned out. Enables components to be re-activated after model instantiation but before running the optimisation problem. Defaults to False for backwards compatibility.

|new| Optimisation problems can be built and solved with the open-source HiGHS solver via a new, direct backend interface: config.build.backend: highs (#856). Piecewise constraints are not yet available in this backend.

|removed| link_flow_cap decision variable, for constraining the flow_cap either end of a transmission link to the same value. Instead, rather than via this auxiliary variable, the constraint is applied directly to each flow_cap entry.

|changed| link_from/link_to and one_way transmission technology parameters persist in the inputs array and their handling is deferred to the math.

|new| map_dim math helper function, to map from a string lookup array indexed over dimension(s) != , where entries are references to values of dimension , to a boolean array. |fixed| "Running in Python" docs to load models with calliope.read_yaml instead of the removed calliope.Model(path) instantiation form (#870).

|fixed| Default values on masked array entries entering into math expression summation. In particular, fixes sum(flow_cap, over=carriers) on the constraint source_availability_supply erroneously returning inf (#862).

|fixed| Time clustering method, to handle cases where floating point errors can lead to index mismatches and dropped clusters (#826).

|new| user-defined math example introducing monthly peak charges (#727).

|changed| data table key data → table and parameters → inputs, to avoid confusion with math components (#814).

|fixed| parameters loaded in from multiple data tables indexed over different dimensions broadcast across nodes and techs correctly (#806).

|new| YAML math section for creating data arrays after solving a model, using expressions combining inputs and/or results (#638).

|new| Dimensions can now be referenced in math where and expression strings and arithmetic can be used when making comparisons and using helper functions in where strings (#733).

|removed| Duplication of math attributes in array attributes. For example, model.inputs.flow_cap_max.attrs["default"] is now only available from model.math.build.parameters.flow_cap_max.default.

|changed| Init config can be updated on re-init and on loading a model from NetCDF, e.g. to apply new subsetting, resampling, clustering, math.

|fixed| Reloading a clustered model from file was attempting to re-cluster (#824).

|fixed| Example notebook outputs no longer clutter search results in docs and search result example notebook pages are grouped under the top-level page header (#675).

|fixed| Documentation now clarifies more explicitly that Calliope is unit-agnostic and that capacity and its costs are relative to carrier flow (#775).

|fixed| Documentation explains better how dimensions and broadcasting work (#667).

|fixed| Documentation improvements: Clarify dimension subsetting documentation, always reference calliope.read_yaml(path) rather than calliope.Model(path) (#829)

|changed| Community and help links now point to Zulip (https://calliope-modelblocks.zulipchat.com/) instead of GitHub Discussions.

|fixed| Pre-defined operate mode math is now documented alongside the other built-in math.

|fixed| Built-in math documentation now shows components that an override deactivates (REMOVED) and those it moves to a different component group, e.g. from decision variable to parameter (REDEFINED).

|changed| Math documentation includes math components that have no math expression to show, documenting them by their description alone.

What's Changed

New Contributors

Full Changelog: https://github.com/calliope-project/calliope/compare/v0.7.0.dev7...v0.7.0.dev8

Energy Systems - Energy System Modeling Frameworks - Python
Published by irm-codebase 2 days ago

Calliope - Release v0.7.0.dev7

User-facing changes

|changed| Model.from_dict has been removed in favor of calliope.read_dict (matching read_netcdf and read_yaml)

|changed| in model.yaml, parameters has been renamed to data_definitions to fit with the new params / lookups approach.

|changed| config.init.broadcast_param_data has been renamed to config.init.broadcast_input_data to fit with the new params / lookups approach.

|new| config.solve.postprocess_active can be used to skip postprocessing to save computation time, or avoid crashes if base math is made incompatible with them.

|fixed| All math components units and default values (#662).

|changed| pre-validation of math strings, in search of parsing errors, is undertaken at model instantiation and defaults to not run to reduce model init time.

|changed| Math dictionaries can be injected on loading a model from YAML/dict when using the Python API (calliope.read_yaml(...)/calliope.Model.from_dict(...)) using the math_dict argument.
This replaces add_math_dict in calliope.Model.build.

|changed| MILP math formulation now has its own file (milp.yaml), which can be requested via extra_math (#749).

|fixed| Updates to the expression underlying the Gurobi backend objective function make their way through to the objective function before solving (#797).

|new| Math global expressions have an order option to allow user-defined math to be reordered relative to pre-defined math (#773).

|changed| |backwards-incompatible| where: now uses == for equality comparisons (previously =) (#728).

|changed| |backwards-incompatible| YAML files are loaded with calliope.read_yaml(...) instead of calliope.Model(...).

|changed| |backwards-incompatible| In-memory dictionaries are loaded with calliope.Model.from_dict(...) instead of calliope.Model(...).

|new| In-memory xarray datasets are loaded with calliope.Model(...), e.g. calliope.Model(inputs=input_array, attrs=attributes).

|changed| Model attributes are not stored in the input/results arrays (e.g. model.inputs.attrs) but are available directly from the model object:

  • model.definition = the loaded model data definition
  • model.config = the loaded model configuration
  • model.runtime = runtime attributes, like timing logs
  • model.math = initialised and applied math

They can all be dumped to a single attribute dictionary with calliope.Model.dump_all_attrs()

|new| Model attributes are saved to a attrs.yaml YAML file when storing data using calliope.Model.to_csv.

|new| helper functions to enable summation over single-/multi-dimension groups and rolling window summations of math expression components (#735, #777).

|fixed| Evaluating results of Gurobi global expressions containing pure decision variables / parameters (#780).

|changed| SOS2 piecewise cost example in docs to explicitly include the new decision variable in the investment cost where string (see #525).

|fixed| Timeseries capacity factor accounts for time resolution (#762).

|fixed| Levelised cost accounts for energy that is generated and exported (flow_export) (#767).

|new| all math is defined during init and then stored in model._def.math, with the option to completely replace plan.yaml as the base mode (#763, #739).

|new| documentation on SPORES-specific configuration options (#750, #752).

|fixed| SPORES mode models not being appropriately serialised on saving to NetCDF (#751, #752).

|new| backend get_objective method, similar to get_global_expression etc. (#761)

|new| arrays in model results containing the objective function value of solved models.
Arrays have the name of the corresponding objective in math.
E.g., model.results.min_cost_optimisation will give the objective function value for the base math objective.

|changed| SPORES mode baseline run renamed to 0 and spores array dimension set to an integer dtype

|changed| SPORES configuration option skip_baseline_run renamed to use_latest_results

|new| config.solve.spores.use_latest_results allows for running from existing baseline (e.g. plan mode) results and for continuing a set of SPORES runs.
E.g., if there are results for 4 SPORES, the run can be continued for 6 more iterations by setting spores.use_latest_results and updating spores.number to 10 (4 + 6).

Internal changes

|fixed| ModelDataFactory is no longer run twice during startup, resulting in faster initialisation

|fixed| SPORES tests to vary capacities of costed technologies rather than be able to simply vary the capacity of a free heat transmission technology between SPORES (#782).

|fixed| Randomly failing tests that rely on random sampling from the core code, by setting a global test suite random seed (#789).

|new| pydantic.RootModel used for storing model definition dictionaries with arbitrary keys (techs, nodes, etc.).

|new| Add rich as a dev dependency for much more readable repr strings of our pydantic models.

|changed| Separate inputs and results into separate datasets rather than filtered views on the same _model_data, private dataset.
Accordingly, Remove is_result array attribute and save to separate NetCDF "groups"

|changed| Removed all attributes from _model_data.attrs, storing them instead in pydantic model objects attached to the main calliope model object.
On saving to file, attributes are stored in the attrs of an empty dataset as a distinct NetCDF "group".

New Contributors

Full Changelog: https://github.com/calliope-project/calliope/compare/v0.7.0.dev6...v0.7.0.dev7

Energy Systems - Energy System Modeling Frameworks - Python
Published by irm-codebase 12 months ago

Calliope - Release v0.7.0.dev6

|new| working SPORES mode, upgraded from v0.6 to include a selection of scoring algorithms (#716).

|new| backend set_objective method, to switch between pre-defined objectives (#716).

|changed| |backwards-incompatible| from and to parameters (to define start and end point of a transmission link) are now link_from and link_to (#717).

|changed| |backwards-incompatible| operate and spores mode configuration options are now nested within the main configuration (e.g., build.operate_window is now build.operate.window and solve.spores_number is now solve.spores.number) (#704).

|changed| coin-or-cbc is now available cross-platform on conda-forge and so is the recommended open-source solver to install a user environment with (#744).

|changed| Upper bound pins for dependencies removed where possible, to minimise clashes when using calliope as a dependency in a project (#744).

|changed| |backwards-incompatible| to ensure the model configuration always remains in sync with the results, kwargs in model.build() and model.solve() now directly affect model.config (#704)

|changed| template: can now be used anywhere within YAML definition files, not just in the nodes, techs and data_tables sections (#719).

|changed| Removed inheritance math helper function since we use base_tech for abstract base technologies and templates are now applied too early to be available later (#719).
To refer to template inheritance, set a parameter within a template as all children will share the same value.

Energy Systems - Energy System Modeling Frameworks - Python
Published by brynpickering over 1 year ago

Calliope - Release v0.7.0.dev5

|changed| "An overview of the Calliope terminology" information admonition to remove self-references and improve understandability.
Now also includes a visual depiction of how the different defined components connect together (#699).

|fixed| Area-based parameters have appropriate documented units of area rather than area^2 (#701).

|fixed| Technology capacity lower bound constraints so that [cap-type]_min (e.g., flow_cap_min) is not always enforced if the purchased_units variable is active (#643).

|changed| Single data entries defined in YAML indexed parameters will not be automatically broadcast along indexed dimensions.
To achieve the same functionality as in <v0.7.dev4, the user must set the new init configuration option broadcast_param_data to True (#615).

|changed| Helper functions are now documented on their own page within the "Defining your own math" section of the documentation (#698).

|new| where(array, condition) math helper function to apply a where array inside an expression, to enable extending component dimensions on-the-fly, and applying filtering to different components within the expression (#604, #679).

|new| Data tables can inherit options from templates, like techs and nodes (#676).

|new| dimension renaming functionality when loading from a data source, using the rename_dims option (#680).

|changed| cost expressions in math, to split out investment costs into the capital cost (cost_investment), annualised capital cost (cost_investment_annualised), fixed operation costs (cost_operation_fixed) and variable operation costs (cost_operation_variable, previously cost_var) (#645).

|new| Math has been removed from model.math, and can now be accessed via model.math.data (#639).

|new| (non-NaN) Default values and data types for parameters appear in math documentation (if they appear in the model definition schema) (#677).

|changed| data_sources -> data_tables and data_sources.source -> data_tables.data.
This change has occurred to avoid confusion between data "sources" and model energy "sources" (#673).

Energy Systems - Energy System Modeling Frameworks - Python
Published by brynpickering over 1 year ago

Calliope - Release v0.7.0.dev4

|fixed| Decision variable domain in math docs to use $\in$ instead of $\forall$ (#652).

|fixed| Clarity of flow_cap_min description in documentation (#653).

|changed| API/schema documentation is de-ranked in documentation search bar results (#670).

|new| Math component cross-references in both directions ("uses" and "used in") in Markdown math documentation (#643).

|fixed| Duplicated links in math documentation (#651).

|changed| node_groups and tech_groups changed to a general top-level templates key,
accessed via the template key (replacing inherit) in nodes and techs (#600).

|fixed| Contribution of cost_om_annual_investment_fraction to total investment costs, to not apply to depreciated costs (#645).

|fixed| Math for multi-carrier variable export costs (#663).

|new| Piecewise constraints added to the YAML math with its own unique syntax (#107).
These constraints will be added to the optimisation problem using Special Ordered Sets of Type 2 (SOS2) variables.

|new| Direct interface to the Gurobi Python API using !#yaml config.build.backend: gurobi or !#python model.build(backend="gurobi").
Tests show that using the gurobi solver via the Python API reduces peak memory consumption and runtime by at least 30% for the combined model build and solve steps.
This requires the gurobipy package which can be installed with mamba: mamba install gurobi::gurobi.

|fixed| Force a header row in tabular data loaded from CSV to (#596).
Fixes issue where unexpected index levels can end up in the loaded data (#573).

|fixed| Single element lists/sets in the model Dataset attribute dictionary are restored to lists/sets on loading from NetCDF (#614).

|new| Decision variables and global expressions can have a title defined, which will be available in the model results as attributes of those components and can be used for e.g. visualisation (#582).
Parameter titles from the model definition schema will also propagate to the model inputs.

|fixed| Backend parameter updates propagate correctly through global expressions in the order those expressions were defined (#616).

|fixed| If setting model.backend.verbose_strings(), rebuilt model components from making backend parameter updates will automatically have verbose strings (#623).

|fixed| Erroneous use of dimensions: in docs example of an indexed parameter (#612).

|changed| add_dimensions to add_dims in data_sources definition to align with dims in indexed parameter definition (#621).

|new| Allow extracting shadow prices into results by listing constraints in config.solve.shadow_prices, e.g. config.solve.shadow_prices: ["system_balance"] Shadow prices will be added as variables to the model results as shadow_price_{constraintname}, e.g. shadow_price_system_balance.

|new| Model stores key timestamps as attributes:

  • timestamp_model_creation: at the start of Model.__init__()
  • timestamp_build_started: at the start of Model.build()
  • timestamp_build_complete: at the end of Model.build()
  • timestamp_solve_started: at the start of Model.solve()
  • timestamp_solve_complete: at the end of Model.solve()

Energy Systems - Energy System Modeling Frameworks - Python
Published by sjpfenninger almost 2 years ago

Calliope - Release v0.7.0.dev3

Minor updates to the v0.7.0 pre-release, mostly to address documentation issues.

|new| mkdocs_tabbed option when writing math documentation to file (calliope.Model.math_documentation.write(...)) which will add YAML snippets to all rendered math as a separate "tab" if writing to Markdown.
Requires the PyMdown tabbed extension to render the tabs correctly in an MkDocs project.

|new| List of pre-defined parameters given in the pre-defined math documentation, with references back to the constraints/variables/global expressions in which they are defined (either in the expression string or the where string).

|new| Units and default values for variables and global expressions added to the math documentation.

|new| Variables and global expressions can have a default value, which is used to fill missing array elements when doing math operations.
These default values ensure that NaN doesn't creep into the built optimisation problem math and are set to values that lead to them having no impact on the optimal solution.

|new| Utility function calliope.util.schema.update_model_schema(...) to add user-defined parameters to the model schema / update existing parameters using YAML schema syntax.
calliope.util.schema.reset() can be used to clean the model schema and return to the original, pre-defined schema.

|fixed| Timeseries clustering file can be a non-ISO standard date format.
Both the index and the values of the timeseries (both being date strings) should be in the user-defined config.init.time_format.

|fixed| the decision variable purchased_units is linked to flow_cap even if neither of the parameters flow_cap_min or flow_cap_max have been defined by the user.

|changed| inbuilt math -> pre-defined math and custom math -> pre-defined math in the documentation.

Energy Systems - Energy System Modeling Frameworks - Python
Published by brynpickering over 2 years ago

Calliope - Release v0.7.0.dev2

v0.7 includes a major change to how Calliope internally operates.
Along with this, there are multiple changes to how Calliope models are defined and configured.
This requires adapting models to work with v0.7.

Due to the scale of these changes, we are staging the development with a series of pre-releases for users to test the functionality.

Below is a summary of the changes made in this pre-release. For more detail on the changes in v0.7 compared to v0.6, see our page on migrating between the two in our documentation.

|new| Storage buffers available in all technology base classes.

|new| Multiple carriers and different carriers in/out available in all technology base classes.

|new| node_groups added to match tech_groups.

|new| technology efficiencies split into inflow and outflow efficiencies.

|new| Technology capacities and efficiencies can be differentiated between technology carriers.

|new| Parameters can be defined outside the scope of nodes and techs using the top-level parameters key in YAML.

|new| Any parameters can be indexed over arbitrary dimensions, both core Calliope dimensions and new, user-defined dimensions.

|new| Non-timeseries data can be loaded from CSV files or in-memory Pandas DataFrames using the top-level data_sources key in YAML.

|new| User-defined mathematical formulations using the new Calliope math syntax can be loaded when creating a model.

|new| Shadow prices obtained from a dual LP problem can be read by using model.backend.shadow_prices.get("constraint_name").

|changed| |backwards-incompatible| Updated to support Python versions >= 3.10.

|changed| |backwards-incompatible| Updated to Pandas >= v2.1, Pyomo >= v6.4, Xarray >= v2023.10.

|changed| |backwards-incompatible| Flat technology definitions, removing the distinction between essentials, constraints and costs.

|changed| |backwards-incompatible| Timeseries data is defined under the data_sources top-level key, not with file=/df= at the technology level.

|changed| |backwards-incompatible| Demand and carrier consumption values are strictly positive instead of strictly negative.

|changed| |backwards-incompatible| model.run() method → two-stage model.build() + model.solve() methods.

|changed| |backwards-incompatible| model and run top-level keys → config.init/.build/.solve.

|changed| |backwards-incompatible| locations top-level key and loc data dimensions → nodes.

|changed| |backwards-incompatible| parent technology parameter → base_tech + inherit.

|changed| |backwards-incompatible| Cost parameters are flattened and use the indexed parameter syntax.

|changed| |backwards-incompatible| links top-level key → transmission links defined in techs.

|changed| |backwards-incompatible| Various parameters/decision variables renamed (namely energy_ → flow_, carrier_prod/_con → flow_out/_in, and resource → source_use/sink_use).

|changed| |backwards-incompatible| Various configuration options renamed.

|changed| |backwards-incompatible| force_resource technology parameter → source_use_equals / sink_use_equals.

|changed| |backwards-incompatible| units + purchased decision variables → purchased_units.

|changed| |backwards-incompatible| Parameters added to explicitly trigger MILP and storage decision variables/constraints.

|changed| |backwards-incompatible| Structure of input and output data within a Calliope model updated to remove concatenated loc::tech::carrier sets and technology subsets (e.g. techs_supply) in favour of sparse arrays indexed over core dimensions only (nodes, techs, carriers, timesteps).

|changed| |backwards-incompatible| coordinates.lat/lon node parameter → latitude/longitude.

|changed| |backwards-incompatible| Distance units default to kilometres and can be reverted to metres with config.init.distance_unit.

|changed| |backwards-incompatible| operate mode input parameters now expected to match plan mode decision variable names (e.g., flow_cap).

|changed| |backwards-incompatible| Cyclic storage is defined per-technology, not as a top-level configuration option.

|changed| Documentation has been overhauled.

|removed| _equals constraints.
Use both _min and _max constraints to the same value.

|removed| x/y coordinates.
Use geographic lat/lon coordinates (in EPSG:4326 projection) instead.

|removed| Comma-separated node definitions.
Inheriting duplicate definitions from node_groups instead.

|removed| supply_plus and conversion_plus technology base classes.
Use supply and conversion technology base classes instead.

|removed| carrier key.
Use carrier_in and carrier_out instead.

|removed| carrier_tiers and carrier_ratios.
Use indexed parameter definitions for flow_out_eff and custom math instead.

|removed| calliope.Model.get_formatted_array.
The Calliope internal representation of data now matches the format achieved by calling this method in v0.6.

|removed| Group constraints.
Use custom math instead.

|removed| Configuration options for features we no longer support.

|removed| Plotting.
See our documentation for example of how to visualise your data with Plotly.

|removed| Clustering.
Cluster your data before creating your Calliope model.
Mapping of timeseries dates to representative days is still possible.

Energy Systems - Energy System Modeling Frameworks - Python
Published by brynpickering over 2 years ago

Calliope - Release v0.6.10

Version 0.6 release compatible with Apple Silicon devices (osx-arm64).

Energy Systems - Energy System Modeling Frameworks - Python
Published by brynpickering over 3 years ago

Calliope - Release v0.6.9

Energy Systems - Energy System Modeling Frameworks - Python
Published by brynpickering over 3 years ago

Calliope - Release v0.6.8

Energy Systems - Energy System Modeling Frameworks - Python
Published by brynpickering over 4 years ago

Calliope - Release v0.6.7

Energy Systems - Energy System Modeling Frameworks - Python
Published by brynpickering about 5 years ago

Calliope - Release v0.6.6-post1

Energy Systems - Energy System Modeling Frameworks - Python
Published by sjpfenninger almost 6 years ago

Calliope - Release v0.6.5

Energy Systems - Energy System Modeling Frameworks - Python
Published by brynpickering over 6 years ago

Calliope - Release v0.6.4

Energy Systems - Energy System Modeling Frameworks - Python
Published by brynpickering over 7 years ago

Calliope - Release v0.6.3

Energy Systems - Energy System Modeling Frameworks - Python
Published by sjpfenninger almost 8 years ago

Calliope - Release v0.6.2

Energy Systems - Energy System Modeling Frameworks - Python
Published by brynpickering over 8 years ago

Calliope - Release v0.6.1

Energy Systems - Energy System Modeling Frameworks - Python
Published by sjpfenninger over 8 years ago

Calliope - Release v0.6.0

Energy Systems - Energy System Modeling Frameworks - Python
Published by brynpickering over 8 years ago

Calliope - Release v0.5.4

Energy Systems - Energy System Modeling Frameworks - Python
Published by sjpfenninger almost 9 years ago

Calliope - Release v0.5.3

Energy Systems - Energy System Modeling Frameworks - Python
Published by brynpickering about 9 years ago

Calliope - Release v0.5.2

Energy Systems - Energy System Modeling Frameworks - Python
Published by sjpfenninger about 9 years ago

Calliope - Release v0.5.1

Energy Systems - Energy System Modeling Frameworks - Python
Published by sjpfenninger about 9 years ago

Calliope - Release v0.5.0

Energy Systems - Energy System Modeling Frameworks - Python
Published by brynpickering over 9 years ago

Calliope - Release v0.4.1

Energy Systems - Energy System Modeling Frameworks - Python
Published by sjpfenninger over 9 years ago

Calliope - Release v0.4.0

Energy Systems - Energy System Modeling Frameworks - Python
Published by sjpfenninger over 9 years ago

Calliope - Release v0.3.7

Energy Systems - Energy System Modeling Frameworks - Python
Published by sjpfenninger over 10 years ago

Calliope - Release v0.3.6

Energy Systems - Energy System Modeling Frameworks - Python
Published by sjpfenninger almost 11 years ago

Calliope - Release v0.3.5

Energy Systems - Energy System Modeling Frameworks - Python
Published by sjpfenninger almost 11 years ago

Calliope - Release v0.3.4

Energy Systems - Energy System Modeling Frameworks - Python
Published by sjpfenninger over 11 years ago

Calliope - Release v0.3.3

Energy Systems - Energy System Modeling Frameworks - Python
Published by sjpfenninger over 11 years ago

Calliope - Release v0.3.2

Energy Systems - Energy System Modeling Frameworks - Python
Published by sjpfenninger over 11 years ago