The Public Utility Data Liberation Project

Makes the US' energy data easier to access and use.
https://github.com/catalyst-cooperative/pudl

Category: Energy Systems
Sub Category: Energy Data Accessibility and Integration

Keywords

cems climate coal ddj eia eia860 eia923 electricity emissions energy epa etl ferc ghg natural-gas open-data pudl python sqlite utility

Keywords from Contributors

archiving climate-change energy-data measur reproducibility policy environmental-data zenodo observation parquet

Last synced: about 16 hours ago
JSON representation

Repository metadata

The Public Utility Data Liberation Project provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists.

• Host: GitHub
• URL: https://github.com/catalyst-cooperative/pudl
• Owner: catalyst-cooperative
• License: mit
• Created: 2017-02-01T17:45:40.000Z (about 8 years ago)
• Default Branch: main
• Last Pushed: 2025-04-21T19:59:09.000Z (5 days ago)
• Last Synced: 2025-04-21T20:26:47.479Z (5 days ago)
• Topics: cems, climate, coal, ddj, eia, eia860, eia923, electricity, emissions, energy, epa, etl, ferc, ghg, natural-gas, open-data, pudl, python, sqlite, utility
• Language: Python
• Homepage: https://catalyst.coop/pudl
• Size: 439 MB
• Stars: 532
• Watchers: 20
• Forks: 121
• Open Issues: 447
• Releases: 23
• Metadata Files:
  • Readme: README.rst
  • Contributing: CONTRIBUTING.rst
  • Funding: .github/FUNDING.yml
  • License: LICENSE.txt
  • Code of conduct: docs/CODE_OF_CONDUCT.md
  • Citation: CITATION.cff
  • Zenodo: .zenodo.json

README.rst

          ===============================================================================
The Public Utility Data Liberation Project (PUDL)
===============================================================================

.. readme-intro

.. |repo-status| image:: https://www.repostatus.org/badges/latest/active.svg
   :target: https://www.repostatus.org/#active
   :alt: Project Status: Active
.. |pytest| image:: https://github.com/catalyst-cooperative/pudl/workflows/pytest/badge.svg
   :target: https://github.com/catalyst-cooperative/pudl/actions?query=workflow%3Apytest
   :alt: PyTest Status
.. |codecov| image:: https://img.shields.io/codecov/c/github/catalyst-cooperative/pudl?style=flat&logo=codecov
   :target: https://codecov.io/gh/catalyst-cooperative/pudl
   :alt: Codecov Test Coverage
.. |rtd| image:: https://img.shields.io/readthedocs/catalystcoop-pudl?style=flat&logo=readthedocs
   :target: https://catalystcoop-pudl.readthedocs.io/en/nightly/
   :alt: Read the Docs Build Status
.. |ruff| image:: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json
   :target: https://github.com/astral-sh/ruff
.. |pre-commit-ci| image:: https://results.pre-commit.ci/badge/github/catalyst-cooperative/pudl/main.svg
   :target: https://results.pre-commit.ci/latest/github/catalyst-cooperative/pudl/main
   :alt: pre-commit CI
.. |zenodo-doi| image:: https://zenodo.org/badge/80646423.svg
   :target: https://zenodo.org/badge/latestdoi/80646423
   :alt: Zenodo DOI
.. |office-hours| image:: https://img.shields.io/badge/calend.ly-officehours-darkgreen
   :target: https://calend.ly/catalyst-cooperative/pudl-office-hours
   :alt: Schedule a 1-on-1 chat with us about PUDL.
.. |mastodon| image:: https://img.shields.io/mastodon/follow/110855618428885893?domain=https%3A%2F%2Fmastodon.energy&style=social&color=%23000000&link=https%3A%2F%2Fmastodon.energy%2F%40catalystcoop
   :target: https://mastodon.energy/@catalystcoop
   :alt: Follow Catalyst Cooperative on Mastodon
.. |slack| image:: https://img.shields.io/badge/Slack-4A154B?logo=slack&logoColor=fff
   :target: https://join.slack.com/t/catalystcooperative/shared_invite/zt-2yg1v2sb7-GsoGlA9Ojc_LCJ00vPWKbQ
.. |linkedin| image:: https://img.shields.io/badge/LinkedIn-0077B5?style=flat&logo=linkedin&logoColor=white
   :target: https://linkedin.com/company/catalyst-cooperative/
   :alt: Follow Catalyst Cooperative on LinkedIn
.. |bluesky| image:: https://img.shields.io/badge/Bluesky-0285FF?logo=bluesky&logoColor=fff&style=flat
   :target: https://bsky.app/profile/catalyst.coop
   :alt: Follow @catalyst.coop on BlueSky
.. |kaggle| image:: https://img.shields.io/badge/Kaggle-20BEFF?style=flat&logo=Kaggle&logoColor=white
   :target: https://www.kaggle.com/datasets/catalystcooperative/pudl-project
   :alt: The PUDL Dataset on Kaggle
.. |youtube| image:: https://img.shields.io/badge/YouTube-%23FF0000.svg?logo=YouTube&logoColor=white
   :target: https://youtube.com/@CatalystCooperative
   :alt: Catalyst Cooperative on YouTube
.. |twitter| image:: https://img.shields.io/badge/X-%23000000.svg?logo=X&logoColor=white
   :target: https://x.com/CatalystCoop
   :alt: Catalyst Cooperative on Twitter
.. |aws| image:: https://img.shields.io/badge/Amazon_AWS-FF9900?style=flat&logo=amazonaws&logoColor=white
   :target: https://registry.opendata.aws/catalyst-cooperative-pudl/
   :alt: PUDL in the AWS Open Data Registry

|repo-status| |pytest| |codecov| |rtd| |ruff| |pre-commit-ci| |zenodo-doi|
|office-hours| |mastodon| |linkedin| |bluesky| |kaggle| |slack| |youtube| |twitter|
|aws|

What is PUDL?
-------------

The `PUDL `__ Project (pronounced puddle) is an open source
data processing pipeline that makes US energy data easier to access and use
programmatically.

Hundreds of gigabytes of valuable data are published by US government agencies, but it's
often difficult to work with. PUDL takes the original spreadsheets, CSV files, and
databases and turns them into a unified resource. This allows users to spend more time
on novel analysis and less time on data preparation.

The project is focused on serving researchers, activists, journalists, policy makers,
and small businesses that might not otherwise be able to afford access to this data from
commercial sources and who may not have the time or expertise to do all the data
processing themselves from scratch.

We want to make this data accessible and easy to work with for as wide an audience as
possible: anyone from a grassroots youth climate organizers working with Google sheets
to university researchers with access to scalable cloud computing resources and everyone
in between!

PUDL is comprised of three core components:

Raw Data Archives
^^^^^^^^^^^^^^^^^
PUDL `archives `__ all our raw
inputs on `Zenodo
`__ to ensure
permanent, versioned access to the data. In the event that an agency changes how it
publishes data or deletes old files, the data processing pipeline will still have access
to the original inputs. Each of the data inputs may have several different versions
archived, and all are assigned a unique DOI (digital object identifier) and made
available through Zenodo's REST API.  You can read more about the Raw Data Archives in
the `docs `__.

Data Pipeline
^^^^^^^^^^^^^
The data pipeline (this repo) ingests raw data from the archives, cleans and integrates
it, and writes the resulting tables to `SQLite `__ and `Apache
Parquet `__ files, with some accompanying metadata stored as
JSON.  Each release of the PUDL software contains a set of DOIs indicating which
versions of the raw inputs it processes. This helps ensure that the outputs are
replicable. You can read more about our ETL (extract, transform, load) process in the
`PUDL documentation `__.

Data Warehouse
^^^^^^^^^^^^^^
The SQLite, Parquet, and JSON outputs from the data pipeline, sometimes called "PUDL
outputs", are updated each night by an automated build process, and periodically
archived so that users can access the data without having to install and run our data
processing system. These outputs contain hundreds of tables and comprise a small
file-based data warehouse that can be used for a variety of energy system analyses.
Learn more about `how to access the PUDL data
`__.

What data is available?
-----------------------

PUDL currently integrates data from:

* **EIA Form 860**:
  - `Source Docs `__
  - `PUDL Docs `__
* **EIA Form 860m**:
  - `Source Docs `__
* **EIA Form 861**:
  - `Source Docs `__
  - `PUDL Docs `__
* **EIA Form 923**:
  - `Source Docs `__
  - `PUDL Docs `__
* **EIA Form 930**:
  - `Source Docs `__
  - `PUDL Docs `__
* **EIA Annual Energy Outlook (AEO)** (a few tables):
  - `Source Docs `__
* **EPA Continuous Emissions Monitoring System (CEMS)**:
  - `Source Docs `__
  - `PUDL Docs `__
* **FERC Form 1** (dozens of fully processed tables, plus raw data converted to SQLite):
  - `Source Docs `__
  - `PUDL Docs `__
* **FERC Form 714** (a few fully processed tables):
  - `Source Docs `__
  - `PUDL Docs `__
* **FERC Form 2** (raw data converted to SQLite):
  - `Source Docs `__
* **FERC Form 6** (raw data converted to SQLite):
  - `Source Docs `__
* **FERC Form 60** (raw data converted to SQLite):
  - `Source Docs `__
* **NREL Annual Technology Baseline (ATB) for Electricity**:
  - `Source Docs `__
* **GridPath Resource Adequacy Toolkit** (partial):
  - `Source Docs `__
  - `PUDL Docs `__
* **US Census Demographic Profile 1 Geodatabase**:
  - `Source Docs `__

High Priority Target Datasets
-----------------------------

If you're interested in any of these datasets, we'd love to integrate them into PUDL.
Get in touch!

* Additional fully processed FERC Form 1 tables, e.g. Transmission & Distribution assets
* Additional high value EIA AEO tables
* `PHMSA Natural Gas Annual Report `__
* `EIA Form 176 `__
  (The Annual Report of Natural Gas Supply and Disposition)
* `FERC Electric Quarterly Reports (EQR) `__
* `EIA Thermoelectric Water Usage `__
* `FERC Form 2 `__
  (Annual Report of Major Natural Gas Companies)
* `MHSA Mines `__

How do I access the data?
-------------------------

For details on how to access PUDL data, see the `data access documentation
`__. A quick
summary:

* `Datasette `__
  provides browsable and queryable data from our nightly builds on the web:
  https://data.catalyst.coop
* `Kaggle `__
  provides easy Jupyter notebook access to the PUDL data, updated weekly:
  https://www.kaggle.com/datasets/catalystcooperative/pudl-project
* `Cloud storage `__
  is populated by our nightly data builds, and is free to access thanks to the `AWS
  Open Data Registry `__.
* `Zenodo `__
  provides stable long-term access to our versioned data releases with a citeable DOI:
  https://doi.org/10.5281/zenodo.3653158
* `The PUDL Development Environment `__
  lets you run the PUDL data processing pipeline locally.

Organizations using PUDL
------------------------

.. _pudl_orgs_start:

This is a partial list of organizations that have used PUDL in their work.
If your organization uses PUDL we'd love to list you here! Please open a
pull request or email us at `[email protected] `__!

* `RMI `__ via both their `Utility Transition Hub `__ and `Optimus financial modeling tool `__
* `The ZERO Lab at Princeton University `__
* `PyPSA-USA `__ an open source power systems model.
* `Energy Innovation `__
* `The PowerGenome Project `__
* `Singularity Energy `__
* `The Open Grid Emissions Initiative `__
* `Win Climate `__
* `The Deployment Gap Model Education Fund `__

.. _pudl_orgs_end:

Contributing to PUDL
--------------------

Find PUDL useful? Want to help make it better? There are lots of ways to help!

* Check out our `contribution guide `__
  including our `Code of Conduct `__.
* You can file a bug report, make a feature request, or ask questions in the
  `Github issue tracker `__.
* Feel free to fork the project and make a pull request with new code, better
  documentation, or example notebooks.
* `Hire us to do some custom analysis `__ and
  allow us to integrate the resulting code into PUDL.

PUDL Sustainers
---------------

The PUDL Sustainers provide ongoing financial support to ensure the open data keeps
flowing, and the project is sustainable long term. They're also involved in our
quarterly planning process. To learn more see `the PUDL Project page on Open Collective
`__.

Gigawatt Tier (≥$25,000/year)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. image:: docs/images/logos/RMI.png
   :width: 600
   :target: https://rmi.org
   :alt: RMI
   :align: center

.. image:: docs/images/logos/GridLab.png
   :width: 600
   :target: https://gridlab.org
   :alt: GridLab
   :align: center

Megawatt Tier (≥$16,000/year)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. image:: docs/images/logos/ZEROLab.png
   :width: 400
   :target: https://zero.lab.princeton.edu/
   :alt: ZERO Lab at Princeton University
   :align: center

Kilowatt Tier (≥$8,000/year)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Become our first `kilowatt tier sustainer `__!

Major Grant Funders
-------------------

Alfred P. Sloan Foundation
^^^^^^^^^^^^^^^^^^^^^^^^^^

.. image:: docs/images/logos/Sloan.png
   :width: 400
   :target: https://sloan.org/programs/research/energy-and-environment/
   :alt: Alfred P. Sloan Foundation Energy and Environment Program
   :align: center

The PUDL Project has been supported by three grants from the `Alfred P. Sloan
Foundation's Energy and Environment Program
`__, in 2019, 2021, and
2024.

National Science Foundation
^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. image:: docs/images/logos/NSF.png
   :width: 400
   :target: https://new.nsf.gov/funding/opportunities/pathways-enable-open-source-ecosystems-pose
   :alt: National Science Foundation Pathways to Enable Open Source Ecosystems (POSE)
   :align: center

The PUDL Project was awarded a grant from the National Science Foundation's `Pathways to
Enable Open Source Ecosystems (POSE) program
`__
(`award 2346139 `__) in 2024.

Licensing
---------

In general, our code, data, and other work are permissively licensed for use by anybody,
for any purpose, so long as you give us credit for the work we've done.

* The PUDL software is released under
  `the MIT License `__.
* The PUDL data and documentation are published under the
  `Creative Commons Attribution License v4.0 `__
  (CC-BY-4.0).

Contact Us
----------

* For bug reports, feature requests, and other software or data issues please make a
  `GitHub Issue `__.
* For more general support, questions, or other conversations around the project
  that might be of interest to others, check out the
  `GitHub Discussions `__
* If you'd like to get occasional updates about the project
  `sign up for our email list `__.
* Want to schedule a time to chat with us one-on-one about your PUDL use case, ideas
  for improvement, or get some personalized support? Join us for
  `Office Hours `__
* `Follow us here on GitHub `__
* Follow us on Mastodon: `@[email protected] `__
* Follow us on BlueSky:  `@catalyst.coop `__
* `Follow us on LinkedIn `__
* `Follow us on HuggingFace `__
* Follow us on Twitter: `@CatalystCoop `__
* `Follow us on Kaggle `__
* `Follow us on YouTube `__
* `Join our Slack `__
* More info on our website: https://catalyst.coop
* Email us if you'd like to hire us to provide customized data extraction and analysis:
  `[email protected] `__

About Catalyst Cooperative
--------------------------

`Catalyst Cooperative `__ is a small group of data wranglers
and policy wonks organized as a worker-owned cooperative consultancy. Our goal is a
more just, livable, and sustainable world. We integrate public data and perform
custom analyses to inform public policy
(`Hire us! `__). Our focus is primarily on
mitigating climate change and improving electric utility regulation in the United
States.

        

Citation (CITATION.cff)

cff-version: 1.1.0
message: "If you use PUDL, please cite it as indicated below."
authors:
  - family-names: Selvans
    given-names: Zane
    orcid: https://orcid.org/0000-0002-9961-7208
  - family-names: Gosnell
    given-names: Christina
    orcid: https://orcid.org/0009-0004-2979-6142
  - family-names: Sharpe
    given-names: Austen
  - family-names: Norman
    given-names: Bennett
  - family-names: Schira
    given-names: Zachary
  - family-names: Lamb
    given-names: Katherine
  - family-names: Belfer
    given-names: Ella
  - family-names: Xia
    given-names: Dazhong

title: "The Public Utility Data Liberation (PUDL) Project"
version: 2025.2.0
doi: 10.5281/zenodo.3404014
date-released: 2025-02-13

---

Owner metadata

• Name: Catalyst Cooperative
• Login: catalyst-cooperative
• Email: [email protected]
• Kind: organization
• Description: Catalyst is a small data engineering cooperative working on electricity regulation and climate change.
• Website: https://catalyst.coop
• Location: United States of America
• Twitter: CatalystCoop
• Company:
• Icon url: https://avatars.githubusercontent.com/u/25487946?v=4
• Repositories: 81
• Last ynced at: 2025-04-24T01:53:10.458Z
• Profile URL: https://github.com/catalyst-cooperative

---

GitHub Events

Total

• Create event: 383
• Release event: 3
• Issues event: 203
• Watch event: 59
• Delete event: 272
• Issue comment event: 425
• Push event: 878
• Pull request review comment event: 734
• Pull request review event: 716
• Pull request event: 300
• Fork event: 14

Last Year

• Create event: 383
• Release event: 3
• Issues event: 203
• Watch event: 59
• Delete event: 272
• Issue comment event: 425
• Push event: 878
• Pull request review comment event: 734
• Pull request review event: 716
• Pull request event: 300
• Fork event: 14

---

Committers metadata

Last synced: 7 days ago

Name	Email	Commits
Zane Selvans	z****s@c****p	2727
Christina Gosnell	c****l@c****p	1085
Austen Sharpe	a****e@g****m	576
e-belfer	e****r@c****p	370
dependabot[bot]	4****]	334
zschira	z****a@c****p	291
Katie Lamb	k****b@c****p	271
bendnorman	b****9@c****u	250
Jan Rous	r****k@g****m	223
ezwelty	e****y@g****m	165
Dazhong Xia	d****a@c****p	154
pre-commit-ci[bot]	6****]	124
alana	a****n@g****m	102
Trenton Bush	t****h@c****p	87
sbwinter	s****r@g****m	78
karldw	k****w	77
PUDL Bot	7****t	63
Austen Sharpe	a****e@A****l	59
Steven Winter	s****1@g****m	58
Austen Sharpe		42
Pablo Virgo	m****x@p****m	41
Alex Engel	a****l	38
Greg Schivley	g****y@g****m	32
grgmiller	g****r@u****u	27
yashkumar1803	y****3@g****m	25
David Stansby	d****y@g****m	10
Kathryn Mazaitis	1****d	7
Nancy Amandi	1****e	6
Kurt Nordback	k****k@p****m	5
Roberto Zanchi	r****i	3
and 20 more...

Committer domains:

• catalyst.coop: 7
• gitter.im: 1
• erictleung.com: 1
• envelope.city: 1
• codacy.com: 1
• samhburns.com: 1
• auner.org: 1
• apptrain.net: 1
• catayst.coop: 1
• austens-mbp.lan1: 1
• zoe.edu: 1
• ucdavis.edu: 1
• pablovirgo.com: 1
• cornell.edu: 1

---

Issue and Pull Request metadata

Last synced: 1 day ago

More stats: https://issues.ecosyste.ms/repositories/lookup?url=https://github.com/catalyst-cooperative/pudl

---

---

---

Package metadata

• Total packages: 3
• Total downloads:
  • pypi: 940 last-month
• Total dependent packages: 1 (may contain duplicates)
• Total dependent repositories: 9 (may contain duplicates)
• Total versions: 50
• Total maintainers: 3

---

Dependencies

Score: 17.658169551612858