The Public Utility Data Liberation Project - PUDL v2025.2.0

This is our regular quarterly release for 2025Q1. It includes updates to all the datasets that are published with quarterly or higher frequency, plus initial versions of a few new data sources that have been in the works for a while.

One major change this quarter is that we are now publishing all processed PUDL data as Apache Parquet files, alongside our existing SQLite databases. See Data Access for more on how to access these outputs.

Some potentially breaking changes to be aware of:

• In the EIA Form 930 – Hourly and Daily Balancing Authority Operations Report a number of new energy sources have been added, and some old energy sources have been split into more granular categories. See Changes in energy source granularity over time.
• We are now running the EPA’s CAMD to EIA unit crosswalk code for each individual year starting from 2018, rather than just 2018 and 2021, resulting in more connections between these two datasets and changes to some sub-plant IDs.

Many thanks to the organizations who make these regular updates possible! Especially GridLab, RMI, and the ZERO Lab at Princeton University. If you rely on PUDL and would like to help ensure that the data keeps flowing, please consider joining them as a PUDL Sustainer, as we are still fundraising for 2025.

See the v2025.2.0 release notes in our documentation for more details.

What's Changed

New & Updated Data

• Integrate EIA 861 2023 final release data by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3911
• Integrate EIA 930 data up through October by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3946
• Add Q3 2024 CEMS data to ETL by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3948
• EIA bulk elec q3 2024 integration by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3951
• Eia860m q3 24 by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3949
• Eia923 q3 24 by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3950
• Eia176 wide table by @davidmudrauskas in https://github.com/catalyst-cooperative/pudl/pull/3590
• EIA 176 wide tables - follow-up fixes by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3978
• Add EIA 176 2023 data by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/4004
• add censusfips metadata by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/4006
• Enable sec10k distribution by @zschira in https://github.com/catalyst-cooperative/pudl/pull/4026
• Integrate EIA 860 multifuel table by @jmelot in https://github.com/catalyst-cooperative/pudl/pull/3988
• Add Q4 2024 CEMS data to ETL by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/4052
• Update Q1 2025 EIA bulk electricity data by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/4051
• Eia860m eia923 q1 2025 by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/4047

Other Changes

• Post v2024.10.0 release errata cleanup by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3917
• Add some links to the VCE RARE data source page. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3921
• Close out PUDL v2024.11.0 release notes by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3961
• Reduce peak memory use of VCE RARE assets by @zschira in https://github.com/catalyst-cooperative/pudl/pull/3959
• Clean up some nightly build infrastructure cruft by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3962
• Fix nested list formatting in CONTRIBUTING by @crd477 in https://github.com/catalyst-cooperative/pudl/pull/3974
• Updates to splink FERC to EIA record linkage notebook by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/3976
• Correct flipped starting/ending balance column rename. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3979
• Add FERC1 validation tests by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3860
• Fix pudl.settings class doc strings by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/3986
• Add parquet download link to data dictionary by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/3984
• Refactor entity matching name cleaner to be more efficient by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/3953
• Fix row count asset check for vcerare by @zschira in https://github.com/catalyst-cooperative/pudl/pull/3993
• Make asset checks run in integration test by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3990
• Make the reconstructable job not take any arguments at all. by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3996
• Add NREL GHA runner WIF setup by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3952
• Add boolean column naming standard to naming_conventions doc by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3998
• Make it possible to pass another sources dict to DataSource by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/4003
• Finish unittest -> pytest conversion by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/4014
• Pass sources to DataSource.from_id() by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/4013
• Acknowledge PUDL Sustainers by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/4025
• Sec distribution fix by @zschira in https://github.com/catalyst-cooperative/pudl/pull/4031
• Add sec10k metadata directly in PUDL by @zschira in https://github.com/catalyst-cooperative/pudl/pull/4035
• Update PUDL bibliography by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/4049
• Change SEC 10-K table schemas to fix FK errors and use quarterly naming. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/4046
• fix: Normalize VCE RARE spellings of great lakes. by @krivard in https://github.com/catalyst-cooperative/pudl/pull/4029
• Update Q1 2025 EIA 930 data by @krivard in https://github.com/catalyst-cooperative/pudl/pull/4054
• Update EIA-EPA crosswalk to include multiple years of crosswalk data by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/4056
• Publish yearly detailed FERC1 accounting tables by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/4057
• Close out release notes for PUDL v2025.2.0 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/4062

New Contributors

• @crd477 made their first contribution in https://github.com/catalyst-cooperative/pudl/pull/3974
• @jmelot made their first contribution in https://github.com/catalyst-cooperative/pudl/pull/3988
• @krivard made their first contribution in https://github.com/catalyst-cooperative/pudl/pull/4029

Full Changelog: https://github.com/catalyst-cooperative/pudl/compare/v2024.10.0...v2025.2.0

Energy Systems - Energy Data Accessibility and Integration - Python
Published by github-actions[bot] 2 months ago

The Public Utility Data Liberation Project - PUDL v2024.11.0

PUDL v2024.11.0 is a regularly scheduled quarterly release, incorporating a few updates to the following datasets that have come out since the special release we did in October.

What's Changed

New & Updated Data

• Integrate EIA 861 2023 final release data by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3911
• Integrate EIA 930 data up through October by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3946
• Add 2024Q3 CEMS data to ETL by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3948
• EIA bulk elec 2024Q3 integration by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3951
• Eia860m 2024Q3 by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3949
• Eia923 2024Q3 by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3950

Other Changes

• Post v2024.10.0 release errata cleanup by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3917
• Add some links to the VCE RARE data source page. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3921
• Close out PUDL v2024.11.0 release notes by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3961

Full Changelog: https://github.com/catalyst-cooperative/pudl/compare/v2024.10.0...v2024.11.0

Energy Systems - Energy Data Accessibility and Integration - Python
Published by github-actions[bot] 5 months ago

The Public Utility Data Liberation Project - PUDL v2024.10.0

This is a special early release to publish the new VCE Resource Adequacy Renewable Energy (RARE) dataset. It also includes final releases of EIA 860 and 923 data for 2023 and the FERC Form 714 data for 2021-2023, which had previously been integrated from the XBRL data published by FERC. See the release notes for more narrative detail.

What's Changed

New & Updated Data

• Extract 714 xbrl by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3822
• FERC 714: transform of hourly demand table (dbf +xbrl) by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3842
• Add source metadata for vceregen by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3887
• Integrate 2023 EIA 923 final release data by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3903
• Extract VCE renewable generation profiles and remove deprecated gsutil from workflows by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3893
• Transform vceregen renewable generation profiles by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3898

Other Changes

• Update release process documentation by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3807
• added description of Data Source Heading by @Nancy9ice in https://github.com/catalyst-cooperative/pudl/pull/3780
• Make an EIA plant part association table with generators by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3774
• Remove record_id_eia foreign key relationship to out_pudl__yearly_assn_eia_ferc1_plant_parts by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3819
• Add PUDL pronunciation to docs and README by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3817
• Fix Excel and CSV column mapping errors by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3820
• Try to fix CodeCov report uploads in pytest workflow. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3827
• If docs build interrupted, don't commit leftover autoapi files by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3826
• added primary key column in data dictionary page by @Nancy9ice in https://github.com/catalyst-cooperative/pudl/pull/3821
• Raise error if FERC1 column renames don't match expected data by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3791
• edited data dictionary description by @Nancy9ice in https://github.com/catalyst-cooperative/pudl/pull/3781
• Fix bugs in allocate_gen_fuel by @grgmiller in https://github.com/catalyst-cooperative/pudl/pull/3690
• update min-max rows for gen-fuel allocation* assets by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3831
• Add pudl usage metrics gcp infrastructure by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/3841
• Increase nightly build VM disk from 80GB to 100GB by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3853
• Adjust superset row limits by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/3843
• Add documentation section to PR template by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3859
• Move dagster modules to pyproject.toml instead of CLI command by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/3865
• added introduction by @Nancy9ice in https://github.com/catalyst-cooperative/pudl/pull/3848
• FERC 714: Fix defensive check by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3869
• added descriptions of the EIA861 data by @Nancy9ice in https://github.com/catalyst-cooperative/pudl/pull/3808
• Integrate final 2023 EIA 860 data by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3871
• Create an asset factory for FERC 1 output tables by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3883
• Migrate where/when we filter for the freshest XBRL data by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3861
• fix rolling fuel cost average bug by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3892
• Add Mapbox key to superset by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/3854
• Update gridpathratoolkit and vceregen metadata by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3897
• Lint GHA workflows in pre-commit. by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3870
• Update the source.py file with new name for vceregen dataset: vcerare by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3907
• Update language in vcerare datasource description by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3909
• Close out release notes and repo metadata for v2024.10.0 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3916

Full Changelog: https://github.com/catalyst-cooperative/pudl/compare/v2024.8.0...v2024.10.0

Energy Systems - Energy Data Accessibility and Integration - Python
Published by github-actions[bot] 6 months ago

The Public Utility Data Liberation Project - PUDL v2024.8.0

What's Changed

New & Updated Data

• Merge EIA861 short-form transform by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3660
• Update expected row counts in service territory tables. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3673
• Add EIA AEO fuel cost projections by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3656
• Integrate 2023 EIA 860 Early Release by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3681
• Integrate 2024 NREL ATB data by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3719
• Update xbrl extraction to use new version by @zschira in https://github.com/catalyst-cooperative/pudl/pull/3710
• Transform FERC-714 load forecast table by @seeess1 in https://github.com/catalyst-cooperative/pudl/pull/3670
• Integrate EIA 923 ER by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3721
• Eia923 er validation test fix by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3734
• Add title to quarterly updates issue template by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3765
• Add Q2 2024 CEMS data by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3769
• Update DOI for NREL ATB to get error corrections by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3778
• Integrate Eia923 Q2 2024 Data by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3768
• Update 2024 Q2 EIA bulk electricity data by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3785
• Integrate 2024 Half 2 data for EIA 930 by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3789
• Integrate Eia860M Q2 2024 Data by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3767
• Update all FERC dataset DOIs by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3790

Other Changes

• Add staging environment by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3666
• Add retry logic to datastore.get_zipfile_resource. by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3658
• SoD Docs Updates by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3672
• Add data-update github label to the release notes by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3683
• Update nightly build script to quietly publish public Parquet outputs. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3680
• Update conda lockfile by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3685
• Update docs to link to .zip not .gz S3 archives by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3686
• Take the most recently reported generator operating date when there's no 70%+ consistently reported date by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3697
• Add organization using PUDL section to readme by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/3671
• Log datasette access IPs by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3669
• Make asset checks matter to integration tests by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3687
• Fix EIA 860m Changelog bugs by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3694
• Move pudl_etl_job_factory back to pudl.etl.cli.py by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/3711
• Automate issue creation for Quarterly Updates by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3709
• Disable rolling avg to fill missing fuel prices in EIA923 FRC table by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3716
• Update allowable not water limited capacity ratio. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3723
• Remove bad respondent ID 2 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3724
• If branch is a fork, skip the release.yml workflow by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3727
• Add direct_support keys in core_eia860__scd_generators_energy_storage as foreign keys to core_eia__entity_generators by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3699
• Update all documentation URLs to point at nightly not latest by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3740
• Remove additional generator from expected gens_eia860 row count by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3743
• Add generator_operating_date to 860M changelog table by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3751
• Add mozilla repo to wif pool by @zschira in https://github.com/catalyst-cooperative/pudl/pull/3753
• Update some dataset lists and date ranges. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3754
• Fix release.yml fork behaviors by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3788
• repaired ferc broken links by @Nancy9ice in https://github.com/catalyst-cooperative/pudl/pull/3787
• Superset deployment by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/3715
• Close out v2024.8.0 release notes. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3801

New Contributors

• @seeess1 made their first contribution in https://github.com/catalyst-cooperative/pudl/pull/3670
• @Nancy9ice made their first contribution in https://github.com/catalyst-cooperative/pudl/pull/3787

Full Changelog: https://github.com/catalyst-cooperative/pudl/compare/v2024.5.0...v2024.8.0

Energy Systems - Energy Data Accessibility and Integration - Python
Published by github-actions[bot] 8 months ago

The Public Utility Data Liberation Project - PUDL v2024.5.0

What's Changed

New Data

• Update EIA Bulk Electricity archive DOI by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3353
• 3313 Q4 2023 eia860 update by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3367
• Add Q4 CEMS data by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3379
• Extract raw 923 Schedule 8 A-D by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3373
• Integrate monthly EIA923 data through November 2023 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3422
• Add EIA Thermoelectric Cooling Water dataset DOI to datastore. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3457
• Transform EIA860 and EIA923 Cooling System Tables by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3405
• Add manual GridPath RA Toolkit renewable profile data source. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3489
• eia860 solar: extract by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3482
• Extract EIA860 Energy Storage tables by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3488
• NREL ATB axtraction by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3498
• Extract EIA 930 data, refactor extractors to handle different date partitions by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3497
• Extract EIA923 energy storage table by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3516
• Transform EIA860 Wind by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3522
• Transform and harvesting eia860 solar table by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3524
• WIP: GridPath RA Toolkit wind and solar generation profiles by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3514
• Transform and harvesting the eia860 Energy Storage table by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3526
• EIA 923 energy storage transform by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3546
• Extract AEO Table 54, with bonus 13/15/20. by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3538
• Transform NREL ATB by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3570
• EIA-930 initial transform by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3584
• Extract Net Summer Electricity Generating Capacity from AEO Table 54 by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3582
• Update EIA Bulk Electricity archive/DOI. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3615
• Add electric sales transformation. by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3613
• Add EPA CEMS 2024Q1 by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3624
• Q1 2024 eia860m eia923 by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3625

Other Changes

• Fix (more) v2024.02.03 release issues by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3346
• Output Parquet files as well as SQLite in PUDL ETL by @zschira in https://github.com/catalyst-cooperative/pudl/pull/3296
• Split monolithic ferc_to_sqlite ops into per-dataset pieces by @rousik in https://github.com/catalyst-cooperative/pudl/pull/3098
• Add a simple test coverage check. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3352
• Add a simple pytest coverage check on workflow_dispatch or merge queue by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3371
• Provide CodeCov token in pytest workflow. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3374
• Update docs + add release template by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3361
• Stop using live DB in unit tests!! by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3377
• Add sec10k metadata to sources by @zschira in https://github.com/catalyst-cooperative/pudl/pull/3378
• Force --no-cov in nightly build by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3382
• Use context managers for opening zipfiles by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/3369
• Update expected row count for EIA tables post 860m quarterly update by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3380
• Skip batch job if build was skipped as a whole. by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3390
• Update nightly build script to distribute parquet by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3399
• Make an EIA860m Changelog table by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3331
• Parametermize adding a column in the FERC1 transform & ensure _correction records end up in the calculation compoent table by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3409
• Simplify pytest-cov configuration. by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3391
• Prototype dagster-pandera integration by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3282
• Fix small plants input table to FERC all plants table by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/3415
• Standardize process for merging tagged commits into persistent branches automatically by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3347
• Restore individual FERC 1 plant output tables. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3417
• Experiment tracking by @zschira in https://github.com/catalyst-cooperative/pudl/pull/3289
• Address loose ends in versioned release mechanics by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3421
• Close out release notes for PUDL v2024.2.6 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3427
• Fix minor issues that arose in v2024.2.6 release by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3432
• Harvest generator operating dates when they're within a year of one another by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3419
• Add RMI beta access to parquet.catalyst.coop by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3434
• Add new citations of Catalyst / PUDL by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3435
• Add BA codes and EIA sector IDs to EIA-860M changelog table by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3442
• Very minor but widespread formatting changes from ruff 0.3.0 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3445
• Get multiple years of EIA 176/191/757A CSV data by @davidmudrauskas in https://github.com/catalyst-cooperative/pudl/pull/3402
• Delete unused try/except Excel read-in method in pudl.extract.excel by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3454
• Update pull_request_template.md to improve full ETL instructions by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3446
• Fix broken links and rendering failure in PR template by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3458
• Add metadata for ATB, EIA 930 and AEO data by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3474
• Add PUDL citation for Grid Strategies load growth report. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3483
• Clean EIA 860 and 923 FGD operation and maintenance data by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3403
• Fix nightly build FK failure by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3491
• Add logline that tells us more about BadZipFile. by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3493
• Add total -> subtotal calculation correction & fix hard-coded plant-in-service table name by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3450
• Fix indent error in nightly builds by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3521
• add two new correction records into plant_in_service table by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3525
• Ferc1 rate base tag updates by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3517
• Schema cleanup by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3529
• Refactor etl/init.py to make adding new modules easier. by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3539
• Attempt to limit _out_ferc714__hourly_demand_matrix concurrency by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/3541
• Manage concurrency of high-memory processes by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3543
• Tag additional assets as high memory usage by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3548
• Rename BA & Utility service territory tables to use conventions by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3552
• Pin ferc-xbrl-extractor<1.4 to facilitate frictionless v5 update by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3566
• Draft of package-level field encoding, applied to EIA by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3558
• Get last non-null value instead of latest XBRL filing. by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3545
• Update expected row counts for FERC 1 tables by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3574
• Create beta access SA's for gridpath and zerolab. by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3577
• Allow beta service accounts to access Parquet bucket by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3586
• Speed up nb-output-clear step in pre-commit by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3591
• Enumerate all AEO table 54 schemas. by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3588
• Fix quoting in hourly parquet deployment command by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3602
• Remove unused resource keys from asset definitions by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3603
• Stop ignoring test directory passed to pytest. by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3610
• Refactor EIA AEO totals checks. by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3606
• Clean up a couple warnings and remove obsolete materialize script. by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3608
• End use sectors generation by fuel type. by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3598
• Always clobber existing outputs in FERC to SQLite conversions by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3622
• Update EIA AEO table description units to be consistent with columns. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3626
• NREL ATB - Stop dropping duplicate values before unstacking by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3630
• Map new EIA plants and utilities with PUDL IDs for 2024Q1 update by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3636
• Breakdown total utility_type and partial in_rate_base in rate base table by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3532
• Update expected MCOE row counts by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3638
• Add template that includes overview/success criteria/tasks by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3640
• Publish FERC1 Rate Base Table by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3641
• Rate base category tweaks by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3647
• Organize the large new data section of release notes by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3652

Full Changelog: https://github.com/catalyst-cooperative/pudl/compare/v2024.02.04...v2024.5.0

Energy Systems - Energy Data Accessibility and Integration - Python
Published by github-actions[bot] 11 months ago

The Public Utility Data Liberation Project - PUDL v2024.2.6

What's Changed

New Data

• Update EIA Bulk Electricity archive DOI by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3353
• 3313 Q4 2023 eia860 update by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3367
• Add Q4 CEMS data by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3379
• Extract raw 923 Schedule 8 A-D by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3373
• Integrate monthly EIA923 data through November 2023 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3422

Other Changes

• Fix (more) v2024.02.03 release issues by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3346
• Output Parquet files as well as SQLite in PUDL ETL by @zschira in https://github.com/catalyst-cooperative/pudl/pull/3296
• Split monolithic ferc_to_sqlite ops into per-dataset pieces by @rousik in https://github.com/catalyst-cooperative/pudl/pull/3098
• Add a simple test coverage check. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3352
• Add a simple pytest coverage check on workflow_dispatch or merge queue by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3371
• Provide CodeCov token in pytest workflow. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3374
• Update docs + add release template by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3361
• Stop using live DB in unit tests!! by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3377
• Add sec10k metadata to sources by @zschira in https://github.com/catalyst-cooperative/pudl/pull/3378
• Force --no-cov in nightly build by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3382
• Use context managers for opening zipfiles by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/3369
• Update expected row count for EIA tables post 860m quarterly update by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3380
• Skip batch job if build was skipped as a whole. by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3390
• Update nightly build script to distribute parquet by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3399
• Make an EIA860m Changelog table by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3331
• Parametermize adding a column in the FERC1 transform & ensure _correction records end up in the calculation compoent table by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3409
• Simplify pytest-cov configuration. by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3391
• Prototype dagster-pandera integration by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3282
• Fix small plants input table to FERC all plants table by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/3415
• Standardize process for merging tagged commits into persistent branches automatically by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3347
• Restore individual FERC 1 plant output tables. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3417
• Experiment tracking by @zschira in https://github.com/catalyst-cooperative/pudl/pull/3289
• Address loose ends in versioned release mechanics by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3421
• Close out release notes for PUDL v2024.2.6 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3427

Full Changelog: https://github.com/catalyst-cooperative/pudl/compare/v2024.02.04...v2024.2.6

Energy Systems - Energy Data Accessibility and Integration - Python
Published by github-actions[bot] about 1 year ago

The Public Utility Data Liberation Project - PUDL v2024.02.05

What's Changed

New Data

• Extract EIA923 emissions control table and add 2022 final release dat… by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3100
• Update CEMS partitions to handle year-quarter files by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3096
• Start EIA-176 pipelines: company data by @davidmudrauskas in https://github.com/catalyst-cooperative/pudl/pull/3227
• WIP: Extract raw PHMSA transmission data (A-D, H, I) by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3235
• Update EIA bulk electricity data archive. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3249
• Extract raw PHMSA distribution and start of transmission data (Table A-D, H, I) by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2932
• Extract raw tables for PHMSA transmission data Part F & G by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3242
• Map PHMSA Natural Gas Transmission Part R columns by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3269
• Map PHMSA Natural Gas Transmission Part S columns by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3262
• Map PHMSA Natural Gas Transmission Part T columns by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3267
• Map PHMSA Natural Gas Transmission Part N-O columns by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3260
• 3243 PHMSA transmission part J by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3266
• Map PHMSA Natural Gas Transmission Part L columns by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3254
• Map PHMSA Natural Gas Transmission Part M columns by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3270
• Map PHMSA Natural Gas Transmission Part Q columns by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3280
• PHMSA: fix to Part Q - fix the column names for the other materials (NOT MILES!) by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3291
• PHMSA transmission part P by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3279
• Clean up and standardize PHMSA raw assets by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3295
• Add EIA Forms 191 and 757 to sources in PUDL metadata by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3304

Other Changes

• Sort DBs Fly Datasette; don't distribute Datasette's metadata.yml by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3106
• Add support for choosing between multiprocess and inprocess executors via cli flag by @rousik in https://github.com/catalyst-cooperative/pudl/pull/2895
• Improve flexibility for publishing options by @rousik in https://github.com/catalyst-cooperative/pudl/pull/2964
• Hide diffs in lock files by default. by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3103
• Clean up PUDL CLI tools; use Click framework by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3086
• Create new issue template for adding a new year of data by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3089
• Reorganize contributing docs + add process description. by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3044
• add data corrections for "sizable minority" utilities by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3078
• Skip slow tests in pre-commit hook by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3132
• Replace ferc714 @multi_asset with asset factory by @rousik in https://github.com/catalyst-cooperative/pudl/pull/3123
• WIP Transition CEMS paritions to year_quarter from year and quarter by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3139
• Table diff tools by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3128
• Refactor calculation of annualized_respondents_ferc714 by @rousik in https://github.com/catalyst-cooperative/pudl/pull/3024
• Include sub-annual updates in annual_updates docs by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3129
• Script to sync a local directory up to a Zenodo DOI by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3127
• Attempt to create sandbox data release in nightly builds. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3158
• Alter output path for the nightly builds by @rousik in https://github.com/catalyst-cooperative/pudl/pull/3157
• Add EPA CEMS concurrency limit to pudl_etl by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/3160
• Remove obsolete Docker data access instructions. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3156
• Feature branch: Rename core + output assets to match new naming protocols by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2818
• Knowledge contribution docs by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3151
• Update EPA CEMS docs for quarterly data, new table name. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3178
• Improve nightly build and deployment logic by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3164
• Clean up nightly build/deploy w/o nightly branch update. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3188
• Update FERC rate base tags with RMI guidance by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3162
• update the 860m doi by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3189
• Update ferc-ferc plant matching with ccai implementation. by @zschira in https://github.com/catalyst-cooperative/pudl/pull/3007
• Tell setuptools_scm to ignore non-version tags by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3193
• Clarify consistency check and lower threshold from 75% to 74% by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3194
• Reduce FERC1 match threshold in test to 85% by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3197
• Update nightly branch after successful build by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3195
• Use digest instead of tag so VM uses right image. by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3206
• It's german. It means "the dev, the" by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3212
• Switch from dev to main/nightly/stable branch structure. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3216
• Suppress excessive numba logs by @zschira in https://github.com/catalyst-cooperative/pudl/pull/3217
• Make CEMS extraction handle new listed year_quarter partitions by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3187
• Tweak metadata and pyarrow schema methods to work for all tables by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3222
• Add --build-only flag to datasette deploy script. by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3231
• Generalize passing args through to flyctl deploy by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3236
• Create directory for local only notebooks by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/3230
• fill in the annual columns with quarterly balances by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3234
• FERC to EIA Entity Matching Refactor by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/3184
• Apply new naming conventions to devtool notebooks by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/3228
• Update gas price upper bound; Enable null-row check for MCOE. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3252
• Tiny fix to make FERC to EIA tests xfail by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/3257
• PHMSA gas extract step for transmission part k by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3258
• Fix dataframe embedder op factory and tune ferc-ferc model by @zschira in https://github.com/catalyst-cooperative/pudl/pull/3247
• Nightly build quality of life improvements by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3287
• Push to prod Zenodo sometimes, fix pypi release flow, update docs by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3292
• Add issue template for nightly build by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3286
• Parameterize the FERC1 transform step what transfers quarterly filed data into annual columns by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3300
• Add retries, and use fsspec to handle GCS by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3299
• Use Google Batch for full ETL runs by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3211
• Hide _out tables and restore metadata in Datasette by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/3226
• Update build ref by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3321
• Rename some straggler assets by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/3294
• Resource field description cleanup by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3283
• Fix nightly builds 2024-01-31 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3329
• Trigger CI on merge group; only trigger integration on merge group. by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3332
• Fix nightly build failure 2024-02-01 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3334
• ensure all the corrections get tags and add the begining of a rate base asset by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3214
• Update release notes for PUDL v2024.02.03 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3342
• Fix v2024.02.03 build issues by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3344
• Fix (more) v2024.02.03 release issues by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3346

New Contributors

• @davidmudrauskas made their first contribution in https://github.com/catalyst-cooperative/pudl/pull/3227

Full Changelog: https://github.com/catalyst-cooperative/pudl/compare/v2023.12.01...v2024.02.05

Energy Systems - Energy Data Accessibility and Integration - Python
Published by github-actions[bot] about 1 year ago

The Public Utility Data Liberation Project - v2023.12.01

What's Changed

• Dbf xbrl mapping by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2088
• eia860m september update by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2079
• integrate the elecrtric energy source dbf & xbrl tbl by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2094
• fix ferc1 record_id validation errors by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2102
• Electric Dispositions Table by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2100
• Use app token for auto-merging bot PRs when CI passes. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2106
• fix table name in record_id test by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2111
• Transform f1 xmssn line by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2103
• Map f1_bal_sheet_cr by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2113
• Utility plant summary by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2105
• Allow Tox v4+ in the dev extras environment. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2117
• Bump ferc-xbrl-extractor version to avoid Arelle locale issue by @zschira in https://github.com/catalyst-cooperative/pudl/pull/2118
• Map f1_elc_op_mnt_expn table by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2114
• Map f1_comp_balance_db table by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2112
• Merge release branch updates into our working dev branch. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2133
• Add the balance_sheet_assets_ferc1 table by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2127
• Xbrl metadata restructuring by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2136
• Add AWS creds to build-deploy-pudl action and copy outputs to s3 bucket by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2137
• Mitigate zenodo dependency in docs build by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2150
• Update to new version of FERC XBRL Extractor by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2151
• Transform f1_dacs_epda by @zschira in https://github.com/catalyst-cooperative/pudl/pull/2143
• Add depreciation_amortization_summary_ferc1 to non-unique record ID's… by @zschira in https://github.com/catalyst-cooperative/pudl/pull/2154
• Integrate income_statement_ferc1 table by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2147
• Move awscli from pudl package to docker image by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2163
• Ferc1 xbrl table release notes by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2157
• Transform f1_bal_sheet_cr by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2134
• Transform f1_retained_erng xbrl + dbf by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2155
• Transform f1_elc_op_mnt_expn by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2162
• Transform f1_accumdepr_prvsn dbf + xbrl by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2119
• Replace lingering transmission_ferc1 w/ transmission_statistics_ferc1 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2178
• Make transform params stricter 2 by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2177
• Validate the raw ferc1 tables in the settings by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2168
• Update & simplify PR template formatting / language. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2181
• Integrate f1_cash_flow FERC1 table by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2184
• Transform electric_plant_depreciation_functional_ferc1 DBF + XBRL by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2183
• Delete old notebooks by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2186
• Transform f1_elctrc_oper_rev by @zschira in https://github.com/catalyst-cooperative/pudl/pull/2192
• Add direct S3 nightly build download links to README. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2199
• Update documentation to refer to archiver and not scrapers or zenodo-storage by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2190
• Notify community-dev channel by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2211
• Transform f1 othr reg liab by @zschira in https://github.com/catalyst-cooperative/pudl/pull/2215
• Add other_regulatory_liabilities_ferc1 to the list of non-unique reco… by @zschira in https://github.com/catalyst-cooperative/pudl/pull/2222
• Change FuelFix for nuclear from mmmbtu to mmbtu by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2233
• Xbrl test speedups by @zschira in https://github.com/catalyst-cooperative/pudl/pull/2229
• Restrict the FERC1 output tables with the PudlTabl's start/end date by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2238
• Fix fuel ferc1 expected values by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2241
• add methods to PudlTabl so it can be serialized and de-serialized (v2) by @arengel in https://github.com/catalyst-cooperative/pudl/pull/2251
• Retain all reported EIA sector codes for harvesting by @knordback in https://github.com/catalyst-cooperative/pudl/pull/2200
• Pin SQLAlchemy<2.0 and allow pip 23 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2268
• Use Workload Identity Federation in GH Actions by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2259
• Retain all EIA sector IDs for harvesting by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2270
• Split EIA extract steps and add field types to dataset_settings by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2263
• Dagster cli wrapper by @zschira in https://github.com/catalyst-cooperative/pudl/pull/2272
• Add design process documentation by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2282
• Remove tables from settings by @zschira in https://github.com/catalyst-cooperative/pudl/pull/2286
• Add more service accounts to Workload Identity Federation by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2273
• Add EIA 176 to sources.py by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2258
• Docs updates for Annual Updates by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2089
• Add electricity_sales_by_rate_schedule_ferc1 table by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2205
• Rework of FERC to EIA logistic regression model by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/2276
• Add fuel allocation release notes by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2308
• Extract 860 EnviroAssoc and EnviroEquip Tables in PUDL by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2281
• Integrate FERC-EIA record linkage into PUDL by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2224
• Update unit tests for allocate net gen by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2297
• Fix google auth error in tox-pytest by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2311
• Ferc to EIA match release notes by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/2313
• Convert FERC1 -> EIA missing ID validation ET[L] to Dagster by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2309
• Update integration tests to work with Dagster ETL by @zschira in https://github.com/catalyst-cooperative/pudl/pull/2299
• Convert epacems_to_parquet command to run dagster asset by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2300
• Add spot fix function/class by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2254
• Update previous balancing_authority_code_eia fixes for plants_eia860 by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2312
• Breakpoints in Dagster by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2322
• Fix balancing_authority_name update in cases where no ba_name_to_code_map() by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2323
• Update doi to point to new epacamd_eia archive by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2316
• Update local cache when using --gcs-cache-path by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2326
• Resolve dev -> dagster merge fixes by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2318
• Configure dagster env vars from settings if not set already by @zschira in https://github.com/catalyst-cooperative/pudl/pull/2332
• Remove code deprecated by dagster by @zschira in https://github.com/catalyst-cooperative/pudl/pull/2341
• Run nightly builds in dagster-world by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2344
• Update s3 bucket urls to use https by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2351
• Add jobs for excluding EPA CEMS assets by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2343
• Merge s3 readme url changes into dev by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2355
• Add boiler-associated attributes from EIA 860 6.2 EnvrEquip tables to ETL by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2319
• Parameterize reconstructable jobs to set loglevel by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2348
• Merge dev into dagster-asset-etl once again by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2353
• Make nightly build message a little cleaner. by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2363
• Correct eia transform doc strings by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2357
• Add EIA 860 EnviroAssoc/EnviroEquip boiler and emissions control attributes by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2364
• Fix inconsistently reported leading zeros in EIA boiler id by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2367
• Consolidate EPA CEMS parquet files by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2354
• Update main as the nightly builds on dev have succeeded. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2374
• Reorder PK of boilers_eia860 to group data like other tables by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2375
• Remove deprecated ferc extract functions by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2369
• Fix Bandit issues by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2381
• Small tweaks from docs PR by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2372
• Move --no-sign-request to the end of the command in the nightly_data_build docs by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2388
• Bump allowed range of Dagster versions to >=1.1,<1.3 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2382
• Rename electric_opex_ferc1 table electric_operating_expenses_ferc1 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2392
• Dagster filter table schemas by @zschira in https://github.com/catalyst-cooperative/pudl/pull/2393
• Dagster ETL documentation by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2306
• Move prime_mover_code back into boiler_fuel_eia923 by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2362
• dev -> dagster-asset-etl merge by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2398
• Rename PUDL_CACHE to PUDL_INPUT by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2401
• Throw error when db is missing table schema by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2410
• Standardize asset group and asset module names by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2411
• Rename check_pudl_fks to pudl_check_fks to align w/ other CLI names by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2416
• Pin grpcio==1.46.1 (arm64 compatible version available on conda-forge) by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2428
• Convert devtools notebooks to use dagster concepts by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2356
• Add sqlite timeout by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2430
• Get PUDL env settings under test by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2424
• Convert FERC-714 ETL to use Dagster by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2421
• Simplify EPA CAMD EIA X-walk output to just read from DB. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2440
• Expand table descriptions. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1910
• Teach Docker about new env vars by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2443
• Dagster asset etl by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2104
• Fix "Make a PR" link in nightly build success message by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2449
• Add new settings file behavior in docs by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2441
• Convert EIA-861 ETL to use Dagster by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2403
• Integrate EIA-923 Annual Environmental Information (Schedule 8C) spreadsheet maps by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2447
• Throw error if database metadata has changed by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2331
• Untangle eia_transform / harvesting / bga multi-asset by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2450
• Solve 2023-03-29 nightly build failure issues by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2469
• Create PudlSQLiteIOManager to accept a Package object by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2466
• Nightly build 2023-03-30 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2473
• Update PUDL from Python 3.10 to 3.11 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2408
• Use enforce_schema() and read-chunking in PudlSQLiteIOManager. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2459
• Add PUDL_INPUT and PUDL_OUTPUT vars to zenodo-cache-sync action by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2476
• Move project metadata & build specs from setup.py to pyproject.toml by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2479
• Parallelize Dagster processing of EPA CEMS by @zschira in https://github.com/catalyst-cooperative/pudl/pull/2472
• Retry on md5 mismatch in GCS by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2488
• Parallelize tests by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2432
• Use FERC XBRL Extractor 0.8.2 to allow pandas 2.0 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2492
• Dagsterize eia tables by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2496
• Create simple SQL view assets by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2445
• Create Dynamic PudlTabl methods by @zschira in https://github.com/catalyst-cooperative/pudl/pull/2498
• Add funding and release notes URLs to the project. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2507
• Add script to set DB schema before running ETL pipeline by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2515
• Add 1:m matches into plant_parts_eia by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2429
• Move ferc eia manual mapping notebook 2 by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2502
• Dagsterize EIA output tables by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2519
• Validate and save csv of all 1:m FERC-EIA matches by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2516
• Revive comdev notify action by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2538
• Ferc output conversion by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2521
• Flesh-out the Sub Plant ID by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2491
• Dagsterize output tables by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2534
• Manage pudl.sqlite schema with alembic by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2523
• Initial refactoring of dbf extraction process by @rousik in https://github.com/catalyst-cooperative/pudl/pull/2536
• Improve dagster docs by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2451
• Cast resource filter values to lowercase by @rousik in https://github.com/catalyst-cooperative/pudl/pull/2562
• Fix income_statement_ferc1 utility_type categorization bug by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2565
• Update to use 2021 version of the epacamd_eia crosswalk by @zschira in https://github.com/catalyst-cooperative/pudl/pull/2566
• Integrate greg M's PM code fix by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2446
• Revert crosswalk to 2018 by @zschira in https://github.com/catalyst-cooperative/pudl/pull/2582
• Use multiple years in crosswalk by @zschira in https://github.com/catalyst-cooperative/pudl/pull/2580
• Emission control table by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2561
• Merge two 'head' migrations together by @zschira in https://github.com/catalyst-cooperative/pudl/pull/2590
• Clean epacamd_eia mismatches before epacems creation by @zschira in https://github.com/catalyst-cooperative/pudl/pull/2593
• Allow JupyterLab v4.0.0 in pudl-dev environment. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2594
• link straight to development setup page by @AppTrain in https://github.com/catalyst-cooperative/pudl/pull/2592
• Fix rouge operational status codes causing foreign key failure by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2602
• Add encoding step to the transform step for the emissions_control_equ… by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2617
• Compare sub-total calculations to total calculations for XBRL explosion by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2615
• migrate calcuation checks into the ferc1 table transformers by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2618
• Add records correcting FERC 1 calculations that are off by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2620
• Boiler cooler stack flue association tables by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2587
• Boostrap for the run-etl action by @rousik in https://github.com/catalyst-cooperative/pudl/pull/2631
• Boiler cooler stack flue table (bb fixes) by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2629
• Update max rows for boils_eia860, plants_eia860, and pu_eia860 tables… by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2634
• Fix job names in docs by @dstansby in https://github.com/catalyst-cooperative/pudl/pull/2641
• add minor inter table calc fixes by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2635
• Add business_model, service_type to sales_eia861 PK by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2637
• Integrate FERC Form 2 dbf formats into ferc_to_sqlite by @rousik in https://github.com/catalyst-cooperative/pudl/pull/2564
• Rename and test XBRL metadata calculations by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2563
• Use PUDL_INPUT not hard-coded data dir in datastore CLI by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2651
• update expected # of plant_in_service rows by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2650
• Fix ferc2 full etl integration test issues by @rousik in https://github.com/catalyst-cooperative/pudl/pull/2652
• Filter FERC714 ETL by year by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2649
• Minor Documentation changes by @AppTrain in https://github.com/catalyst-cooperative/pudl/pull/2598
• Add release notes explaning FERC 1 metadata cleanup by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2626
• Manage duplicate PKs in EIA-861 table transforms by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2648
• Nightly build 2023 06 14 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2668
• Replace [email protected] with github discussion link on datasette by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2665
• Throw different exception when dbc file is missing by @rousik in https://github.com/catalyst-cooperative/pudl/pull/2654
• Read EIA860 data in parallel by @dstansby in https://github.com/catalyst-cooperative/pudl/pull/2644
• Deduplicate FERC 2 respondent IDs by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2661
• Integrate FERC Form 6 from dbf by @rousik in https://github.com/catalyst-cooperative/pudl/pull/2595
• Fix retained_earnings_ferc1 transform by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2645
• Tweaks to emissions control equipment table by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2664
• Upgrade to ferc-xbrl-extractor v0.8.3 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2675
• Fix Alembic migration diversions in dev by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2681
• Add publish_destinations field to the ETL config by @rousik in https://github.com/catalyst-cooperative/pudl/pull/2659
• Convert EIA-861 and FERC 714 service territory outputs to Dagster assets by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2550
• Publish ferc6.sqlite to Datasette. Add AWS download link. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2686
• Spot fix ferc exploder by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2647
• Bring the Census DP1 to SQLite ETL into dagster by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2621
• Add electric_plant_depreciation_changes_ferc1 into explosion by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2662
• Add electric_plant_depreciation_functional_ferc1 into the calc checking process by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2687
• Update plant_in_service_ferc1 expected row count by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2690
• Allow setuptools 68.0.0 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2692
• Fix apparent typos in FERC 1 rename params. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2691
• Remove [email protected] from docs by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2670
• ferc1 💥 rate base tags by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2697
• fix straggler problems with the rightsutility rename by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2704
• Pin grpcio to 1.56 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2702
• Fix xbrl metadata renaming in the electric_plant_depreciation_functional_ferc1 table by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2712
• Limit dagster concurrency in nightly builds by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2713
• utility_plant_summary_ferc1 to plant_in_service_ferc1 link by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2715
• add straggler dbf-only factoids into ferc1 table metadata by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2716
• Convert EIA generation and fuel allocations to Dagster by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2527
• Remove refs to 2i2c JupyterHub. Move nightly build links to data access page by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2719
• Link electric OpEx to income statement table by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2723
• Identify and link inter-table relationships when they occur within a dimension (e.g. utility_type) by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2669
• Organize FERC 1 XBRL metadata with a calculation forest by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2653
• delete it all!!!! lol remove the no-longer needed duplication removals by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2724
• Fix duplicated components in balance sheet liabilities FERC1 table by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2727
• Make a calculation component table and use in the inter-table/inter-dimension checking by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2721
• Extract FERC Form 60 DBF data to SQLite by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2734
• Add old FERC 60 DBF data to the Datasette deployment. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2739
• Fix small typos by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2725
• Replace references to dagit with dagster UI and dagster-webserver by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2749
• Inject missing dbf-only factoids into XBRL metadata by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2747
• Add parent dimensions into calculation component table by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2753
• Add FERC60 to data access docs by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2750
• Don't drop leaf calculation components from calc comps table. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2754
• Clean-up XBRL calculation fixes by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2728
• Modernize importlib usage by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2759
• Add EIA860, EIA860m, & EIA923 2022 early release data by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2741
• Fix validation errors for 2022 EIA data by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2778
• Dagsterize MCOE output tables by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2553
• Eia861 2022 by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2782
• Simplify config reading and path configurations by @rousik in https://github.com/catalyst-cooperative/pudl/pull/2640
• Fix pudl_setup pudl_in and pudl_out args by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2796
• Apply new naming convention to raw and core intermediate assets by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2789
• Fix anonymous constraints by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2795
• Integrate 2022 CEMS data by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2779
• Use all dimensions in XbrlCalculationForestFerc1 and exploded tables by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2763
• Functional deprish fix by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2794
• Stop importing urllib3 Retry from deprecated location by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2806
• Allow a mix of Zenodo sandbox & production DOIs by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2798
• Update PUDL to pandas 2.0 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2320
• enable FERC explosion tags to be dimension specific by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2817
• Add github action usage notebook and use large runner for tox-pytest … by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2823
• Add Hawaii to CEMS by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2816
• Update description for small plants table by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2815
• Add references to CEMS and Dagster in the annual updates docs by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2814
• Add generator to the retirement_year / _month columns for the monthly… by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2835
• Simplify & enhance linting by using ruff. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2824
• Fix docker builds by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2837
• Skip integraton tests on draft PRs by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2839
• Convert the FERC exploded forest to a table for readability by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2832
• Add assertions and workflow updates to debug ogr2ogr failure. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2849
• Add type hints to helpers and update DBF extraction tests by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2841
• Fix missing keywords in archived datasources by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2851
• Rename s3 bucket by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2793
• Expand multi-dimensional totals correctly by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2855
• Update docs to reflect switch from flake8 to ruff by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2859
• Add contributors & ORCIDs to metadata by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2809
• Remove libabseil version pins since gdal & geopandas have new versions. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2866
• Make leafy balance sheet assets & liabilities data by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2805
• 💥 FERC feature branch 💥 : FERC tables post caclulation validation by concat-ing & deduping by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2633
• Add PHMSA Natural Gas annual report DOI to datastore. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2884
• Update expected row counts in 2 altered FERC 1 tables. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2885
• preliminary version of standardization of the calc metrics by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2880
• Fix inter-table 1:1 dimensions in calculations for XBRL explosion by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2890
• Dagsterize mega_generators and plant_parts_eia by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/2714
• Resolve conflicting dependencies by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2900
• Fix for harvesting owner utilities by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/2903
• remove duplication of explosion input table names by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2921
• Clean up explosion: add plant function tag by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2916
• standardize the calc checks for the total to subtotal calcs by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2886
• Extract more data from FERC XBRLs and handle that new data in ETL by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2821
• Remove libsnappy binary dependency by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2923
• Pin transitive dependency croniter<2; bump pyarrow to v13 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2928
• Fix too-many-paths error in nightly build by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2931
• Unpin croniter as package metadata has been fixed. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2933
• Fix small plant_id_ferc1 fail by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2935
• New zenodo api bandaid by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2942
• dagsterification of ferc1_eia by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2938
• Parallelize extraction of Excel spreadsheets by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2943
• Hotfix docker build by setting LD_LIBRARY_PATH by @rousik in https://github.com/catalyst-cooperative/pudl/pull/2950
• Bump versions for ruff & black by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2952
• FERC1 2022 report year fix by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2947
• FERC1 2022 by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2948
• Trivial change by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2982
• Increase datasette cloud run memory from 4GB to 32GB by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2990
• Improve calculation error checking by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2915
• Only generate alphanumeric entity IDs in test by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2993
• Add data maturity for 923m by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2936
• Fix calculating the report_date in demand_hourly_pa_ferc714 by @rousik in https://github.com/catalyst-cooperative/pudl/pull/2999
• Update sources, DOI and copyright link in PUDL by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/3004
• Switch regex strategy to sampling strategy to improve performance by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/2998
• add alembic schema changes for the recent constraint. by @rousik in https://github.com/catalyst-cooperative/pudl/pull/3012
• Set up Cloud SQL Postgres database for dagster storage by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2996
• Add dagster postgres env vars to build-deploy-pudl.yaml by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/3014
• only fix a reporting_frequency_code when the column exists by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3013
• update excepted rows for no-fips id-ed respondents but keep annualize… by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3023
• Fix validation test_fbp_ferc1_mismatched_fuels error by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3025
• Deploy Datasette to fly.io instead of Cloud Run by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3018
• Test that DB schema matches the Alembic migrations by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3027
• Fix XBRL extraction clobber by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3026
• Always use tmp path for clobber tests. by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3043
• Set up reproducible Python environments with conda-lock by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2968
• Add EIA860 2022 final release data by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3040
• Eia861 2022 final release by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/3048
• Bring conda-lock workflows into main by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3049
• Spot fix cliffside capacity by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/3046
• Always take lockfile version from current branch when merging by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3056
• Update Lockfile by @github-actions in https://github.com/catalyst-cooperative/pudl/pull/3057
• Update PUDL to SQLAlchemy 2.0 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2267
• Pin conda-lock<2.5.0 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3063
• Change lock file merge strategy to look for .yml instead of .yaml files by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/3065
• Fix full build notification logic by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/3058
• Unpin conda lock by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3069
• Use ruff instead of black for autoformatting by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3060
• Update development setup documentation by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3074
• Simplify process of bootstrapping pudl-dev conda env. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3080
• Make nightly build outputs smaller by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3084
• Fix the override paths when running in github actions by @rousik in https://github.com/catalyst-cooperative/pudl/pull/3045
• Clean up explosion by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2894
• Add DBF metadata to electric_plant_depreciation_functional_ferc1 by @e-belfer in https://github.com/catalyst-cooperative/pudl/pull/2918
• Update EIA 860M and EIA 861 DOIs by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3085
• Refactor PUDL to use Pydantic v2 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3051
• Resolve core dump failure by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/3090
• dotenv compatibility by @jdangerx in https://github.com/catalyst-cooperative/pudl/pull/3092
• Eia923 2022 final release q4 update Nov 21 by @robertozanchi in https://github.com/catalyst-cooperative/pudl/pull/3073
• Dev environment cleanup and documentation tweaks by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3093
• Rename dbf-derived FERC SQLite DBs by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3094
• Add GHA workflow for release-on-tag by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/3124

New Contributors

• @knordback made their first contribution in https://github.com/catalyst-cooperative/pudl/pull/2200
• @AppTrain made their first contribution in https://github.com/catalyst-cooperative/pudl/pull/2592
• @dstansby made their first contribution in https://github.com/catalyst-cooperative/pudl/pull/2641
• @robertozanchi made their first contribution in https://github.com/catalyst-cooperative/pudl/pull/3073

Full Changelog: https://github.com/catalyst-cooperative/pudl/compare/v2022.11.30...v2023.12.01

Energy Systems - Energy Data Accessibility and Integration - Python
Published by github-actions[bot] over 1 year ago

The Public Utility Data Liberation Project - PUDL v2022.11.30.post1

This post-release pins our transitive dependency on the h3-py library, which was causing segmentation faults on Apple arm64 architectures in our conda-forge distribution.

For background see:

• Issue https://github.com/catalyst-cooperative/pudl/issues/2426 from @arengel
• PR https://github.com/catalyst-cooperative/pudl/pull/2500 from @nelsonauner
• Upstream issue: https://github.com/uber/h3-py/issues/313
• Attempt to fix in @conda-forge packaging: https://github.com/conda-forge/h3-py-feedstock/pull/39

Energy Systems - Energy Data Accessibility and Integration - Python
Published by zaneselvans about 2 years ago

The Public Utility Data Liberation Project - PUDL v2022.11.30

See the release notes for v2022.11.30 in our docs for a more narrative explanation of what has changed.

What's Changed

This is all the PRs that were merged since the last release, excluding those made by the @dependabot and @pre-commit-ci bots.

• Apply black autoformatting by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1543
• Apply black formatting by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1548
• Update to pip 22, setuptools 61. Add nbconvert to pudl-dev by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1565
• Add installation_year and construction_year to PPL by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/1554
• Modify EPA CEMS ETL to facilitate Intake Catalog by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1563
• Hub EIA transition by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1575
• Add office hours scheduling links to README by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1582
• Rename tox virtualenv dir from .env_pudl to .env_tox by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1586
• Add rstcheck to our collection of linters by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1587
• Bring in year_state_filter tests & improvements from pudl_catalog by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1589
• Update maximum allowed version of setuptools to 62. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1590
• Use partial function in map of EPA CEMS ETL by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1591
• Additional code formatting/linting without modernizing Python syntax by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1598
• Refactor labeling of true granularities with plant part to generator match function by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/1447
• Re-gigger backfilling technology_description & make prime_mover_code an annually harvested column by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1600
• Remove some seldom used dependencies from pudl-dev environment.yml by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1615
• Fix breakage resulting from dask v2022.4.2 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1618
• Cinco de Mayo 🇲🇽 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1616
• Clean operational_status_code using metadata encoder by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1624
• add opex_nonfuel column to all FERC1 plant tables in output layer for all_plants_ferc1 table by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1626
• Add installation_year and construction_year as plant part level by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/1578
• Add ML for sustainable energy citation to bibliography by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1641
• Dependabot auto merge by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1655
• Small docs updates by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1642
• Dynamically generate RSTs with new DataSource metadata by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/1532
• Add DataSource Metadata for EPA-EIA Crosswalk by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1676
• Address issue where 861 ETL fails w/o all years of data by @arengel in https://github.com/catalyst-cooperative/pudl/pull/1671
• update ferc-eia glue with fixes found from the FERC plant-ID-er by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1678
• Rework clean_merge_asof func by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/1550
• Switch to the Furo Sphinx theme by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1680
• Require Python 3.10 and update to modern syntax by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1685
• Release notes for date_merge and default columns change for PPL and MCOE by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/1690
• Apply Yaml pre-commit formatter by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/1689
• Fix all plants ferc1 by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1656
• Require Python 3.10 in the pudl-dev conda environment by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1697
• Draft: add capacity mw to mcoe defaults and fix row counts in validation tests for eia tables by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/1695
• GCE Deploy by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1627
• Add workflow_dispatch support to nightly builds by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/1702
• Add build-deploy-pudl.yml to main by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/1703
• Fix Github Ref bug by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/1704
• Unpin apt-get packages by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/1725
• Remove unnecessary packages from build system; specify backend. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1743
• Update setuptools numpy by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1745
• Add gcs and bypass cache args to datastore cli by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/1740
• Fix dependabot automerge by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1753
• Bring new bot-auto-merge workflow into main by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1756
• Move slowly varying plant attributes from entity to annual plants table by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1749
• Fix bug in gens mega and plant part list creation by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/1759
• Integrate EPA CEMS hourly emissions data for 2021 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1778
• EIA923 early release, EIA860 early release, and 860m 2022-06 by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1834
• Avoid using Shapely v1.8.3 due to upstream bug / incompatibility by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1848
• Update eia923 raw inputs to include revisions made by EIA on 2022-08-11 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1846
• Add missing columns and update EIA860, EIA860M and EIA923 data for 2021 by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1836
• Use gcs cache in ci by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1858
• Patch nightly build flakiness by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/1856
• Create a data_maturity label for EIA data by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1855
• Update bug report issue template by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1869
• Update ETL settings files to work with XBRL+DBF and new Ferc1Settings by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1886
• Xbrl steam but really by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1881
• Use internal zenodo-cache bucket for nightly builds by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/1880
• Encode balancing authority codes by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1897
• Fill in some null BA codes using BA names by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1906
• Implement drop_invalid_rows() for fuel_ferc1 table by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1903
• Split TableTransformer.transform() into 3 phases by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1900
• Prepare raw FERC XBRL DB's for publication with Datasette by @zschira in https://github.com/catalyst-cooperative/pudl/pull/1831
• Aggregate data_maturity in gfn_eia923, update EIA ETL debugging Notebook by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1915
• Use provision-micromamba and remove ferc1_solo ETL to speed up CI. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1913
• Fill in pre-2013 BA codes by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1911
• Update the name of the EPA CAMD to EIA crosswalk data source. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1918
• Add metadata & DOIs for EIA Bulk Electricity data source by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1922
• Integrate EIA-861 2021 Early Release data by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1921
• Updating 861 package_data for 2021 early release by @arengel in https://github.com/catalyst-cooperative/pudl/pull/1920
• Add epacems crosswalk to etl by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1692
• Re-add and update the epacamd-eia crosswalk analysis module by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1934
• Add updated crosswalk analysis back into dev by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1938
• Ensure PUDL works with Pandas 1.5.0 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1902
• Plant part updates to fix RMI CI memory issues by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/1865
• Fix build error for epacamd_eia_test by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1940
• Create github action to sync zenodo caches by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/1935
• Rollback pandas-1.5 by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/1945
• Automate docstring formatting by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1948
• New FERC 1 / XBRL Transform Unit Tests by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1919
• Add a static table of political subdivision attributes. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1966
• Add release notes about the new political_subdivisions table. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1967
• Integrate xbrl to dbf utility ID mapping by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1931
• Refactor FERC 1 transform; integrate XBRL for fuel & steam tables by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1721
• Remove derelict docker build workflow file by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2002
• Add labeling functions for manual maps by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1988
• EIA API Replacement by @TrentonBush in https://github.com/catalyst-cooperative/pudl/pull/1998
• EIA Bulk Electricity Aggregates by @TrentonBush in https://github.com/catalyst-cooperative/pudl/pull/1937
• Add 2021 861 to output by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1990
• Integrating 2021 FERC1 XBRL hydro table by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1992
• Document new transforms & organize imports/logging by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1962
• Add release notes about EIA API removal / fuel price filling by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2008
• Remove EXPERIMENTAL from datasette title by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2010
• Boiler Fuel Allocation Improvements by @grgmiller in https://github.com/catalyst-cooperative/pudl/pull/1608
• Bump allowed geopandas & pygeos conda versions by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2019
• Add 2021 eia923 eia860 final release by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2017
• Docs housekeeping by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2039
• Add a draft PR template. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2038
• Fix erroneous max row values in mcoe_test by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2041
• Allocate fuel consumption by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1096
• Integrate pudl ID's from eia final updates into the xbrl branch by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2042
• Updated docs to include nightly build failure etiquette. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2044
• remove pesky rounds on floats by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2052
• xbrl and dbf integration of pumped storage ferc1 table by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2005
• xbrl and dbf integration of purchased power ferc1 table by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2011
• First draft of plant_in_service_ferc1 DBF + XBRL transform by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2025
• Stop warning about duplicate record IDs & missing columns that don't matter by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2054
• track down and fix warnings in xbrl transform by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2057
• Fix bad multi-index construction that was scrambling XBRL columns. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2059
• Transforms for small gens table XBRL by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2035
• Integrate XBRL taxonomy metadata into plant_in_service transform by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2058
• add release notes for xbrl integration by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2063
• Xbrl nov archive by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2065
• Fix doc stings for small gens table transform by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/2068
• Resolve duplicate DBF-XBRL mappings and duplicate records in plant_in_service_ferc1 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2067
• accumulation of any changes in the xbrl validation tests by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/2071
• Initial FERC Form 1 XBRL Integration by @zschira in https://github.com/catalyst-cooperative/pudl/pull/1665
• Address missing fuel ferc1 records by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2092
• Add loglevel and logfile to all CLI commands by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2091
• Add FERC XBRL DBs to datasette deployment. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2095
• Release 2022.11.XX by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1681
• Add GCP region/zone env var to nightly build scripts by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/2101
• Packaging & release note tweaks for PUDL v2022.11.30 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2098
• fix missing generator_id by @grgmiller in https://github.com/catalyst-cooperative/pudl/pull/2122
• v2022.11.30 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/2135

New Contributors

• @arengel made their first contribution in https://github.com/catalyst-cooperative/pudl/pull/1671
• @grgmiller made their first contribution in https://github.com/catalyst-cooperative/pudl/pull/1608

Full Changelog: https://github.com/catalyst-cooperative/pudl/compare/v0.6.0...v2022.11.30

Energy Systems - Energy Data Accessibility and Integration - Python
Published by zaneselvans over 2 years ago

The Public Utility Data Liberation Project - PUDL v0.6.0

See the more extensive narrative release notes in our documentation.

What's Changed

• Fix release notes formatting and tox -e release warning by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1346
• Minor changes associated w/ data release for v0.5.0 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1348
• Widen allowable Jinja versions to 2-3 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1360
• Bb fips fix by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1364
• Allow PUDL SQLite DB to be loaded into Postgres by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1361
• Add support for Python 3.10 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1373
• Better preserve dtypes in allocate_net_gen process by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1370
• Fill missing technology_description values in generators_eia860 by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1075
• Merge dev into main before 2021-12-20 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1375
• Constrain setuptools to <60.0.0 in environment.yml and pyproject.toml to avoid breaking changes by @Wheelspawn in https://github.com/catalyst-cooperative/pudl/pull/1384
• Implement PyArrow schemas in Pydantic metadata classes by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1377
• Use pd.NA where appropriate for ENUM and categorical fields by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/1376
• Separate resource definitions by data source by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1386
• Export code static labeling to documentation by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/1388
• Replace simple label substitutions with coding tables by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1416
• Remove COLUMN_DTYPES and switch to field metadata dictionary by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1408
• Add test for EIA generator technology_description backfilling by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1389
• Fix county FIPS codes string type in fuel_receipts_costs_eia923 by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/1405
• Add pudl id mapping rst docs by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1387
• Minor changes to make pandas 1.4.0 work by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1421
• Correct time interval in etl_fast.yml description and correct typos in data_access.rst and intro.rst documentation by @Wheelspawn in https://github.com/catalyst-cooperative/pudl/pull/1428
• Address geopandas deprecation warnings by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1444
• Update numba to v0.55 which works w/ Python 3.10 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1449
• Update ci-environment.yml to match pudl-dev environment.yml by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1450
• Consolidate data source metadata using a Pydantic model by @zschira in https://github.com/catalyst-cooperative/pudl/pull/1446
• Clean up FIPS codes and use same method for ZIP codes by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1476
• Fix a few incorrectly mapped PUDL IDs by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1458
• Fix mismapped wheeling power company by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1480
• Valentines Day Merge ❤️ 💞 💘 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1445
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/catalyst-cooperative/pudl/pull/1482
• Customize CodeCov configuration file by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1481
• Use fixed random seeds for timeseries cleaning tests by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1483
• Attempt to resume notifying Slack on CI failures by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1484
• Implement DataSource to raw datapackage method by @zschira in https://github.com/catalyst-cooperative/pudl/pull/1475
• Remove lingering data package refs in docs by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1489
• Propagate Pydantic settings classes throughout the entire ETL by @zschira in https://github.com/catalyst-cooperative/pudl/pull/1506
• Enrich Datasette with new metadata by @katie-lamb in https://github.com/catalyst-cooperative/pudl/pull/1479
• Integrate eia860m data through 2021-12 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1510
• Recombine nuke/non-nuke gen fuel in output functions by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1518
• Move the EIA plant-parts list into PUDL outputs by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1157
• Update release notes with changes since v0.5.0 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1524
• Adjust expected row counts for eia860m-2012-12 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1528
• Remove prefect dependency by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1529
• Potential v0.6.0 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1526

New Contributors

• @Wheelspawn made their first contribution in https://github.com/catalyst-cooperative/pudl/pull/1384
• @katie-lamb made their first contribution in https://github.com/catalyst-cooperative/pudl/pull/1376
• @zschira made their first contribution in https://github.com/catalyst-cooperative/pudl/pull/1446
• @pre-commit-ci made their first contribution in https://github.com/catalyst-cooperative/pudl/pull/1482

Full Changelog: https://github.com/catalyst-cooperative/pudl/compare/v0.5.0...v0.6.0

Energy Systems - Energy Data Accessibility and Integration - Python
Published by zaneselvans about 3 years ago

The Public Utility Data Liberation Project - PUDL 0.5.0

Update to include 2020 annual data

See the more extensive release notes in our documentation.

Merged Pull Requests

• make generation allocation output mirror the standard generation table. by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1134
• Dependency and data release script updates by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1135
• Dependencies by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1150
• End of sprint merge of dev by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1158
• Epic template. by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/1164
• Hourly state demand by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1175
• EIA860 2001-2003 by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/1122
• Redesign metadata and harvest process by @ezwelty in https://github.com/catalyst-cooperative/pudl/pull/806
• Basic epa cems output by @TrentonBush in https://github.com/catalyst-cooperative/pudl/pull/1227
• Map small gen pudl ids by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1231
• Dev PR for sprint ending 2021-09-24 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1228
• Build all generated documentation dynamically by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1235
• Update dependencies, mostly related to testing, plus sklearn 1.0. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1236
• Eliminate null values in generation_eia923 primary key fields by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/1248
• Drop rows with null generator_id in ownership_eia860 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1258
• Add FERC1 output table that combines key FERC1 subtables by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1209
• Deduplicate and re-organize metadata from constants.py by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1230
• Fix utility_id_eia issues in ownership & plants tables by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1268
• remove the data package cruft by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1267
• Updated xlsx_maps for eia860 2020 data by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/1273
• 2020 ferc1 by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1274
• Defer validation of PudlTabl datastore to eia861/ferc714 ETL methods by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1275
• 2020 Harvest and load by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/1277
• Crosswalk analysis by @TrentonBush in https://github.com/catalyst-cooperative/pudl/pull/1256
• Beginnings of a PUDL bibliography by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1294
• add plant_id_pudl to small generators field by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1293
• Deduplicate natural key fields of generation_fuel_eia923 by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/1296
• Integrate 2020 data for ferc1, eia860, eia923 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1297
• Respond to CG's PR comments. Mostly docs. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1308
• EIA-861 FERC-714 2020 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1309
• Boiler fuel duplicate aggregation by @TrentonBush in https://github.com/catalyst-cooperative/pudl/pull/1306
• Fix errors with EIA861 output tables by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1312
• Add missing output tables to EIA861 by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1313
• 2020 Data Integration by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1255
• Update to flake8 v4.0; always install pudl for Tox by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1322
• Use pydantic for ETL settings validation by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/1292
• Update generation_fuel_eia923 documentation with nuclear unit change. by @bendnorman in https://github.com/catalyst-cooperative/pudl/pull/1323
• fix pandas API deprecation (issue #1173) by @TrentonBush in https://github.com/catalyst-cooperative/pudl/pull/1332
• Static metadata tables and automatic recoding by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1272
• Validate v0.5.0 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1345
• PUDL v0.5.0 release candidate by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1334

New Contributors

• @bendnorman made their first contribution in https://github.com/catalyst-cooperative/pudl/pull/1164 🎉

Full Changelog: https://github.com/catalyst-cooperative/pudl/compare/v0.4.0...v0.5.0

Energy Systems - Energy Data Accessibility and Integration - Python
Published by zaneselvans over 3 years ago

The Public Utility Data Liberation Project - PUDL 0.4.0

This is our first release in more than a year and a half, and it contains lots of new data and analyses (and breaking changes...) but it doesn't yet include 2020 datasets for FERC and EIA.

See the complete v0.4.0 release notes for details.

Merged Pull Requests

• Unified logic for excel extraction by @rousik in https://github.com/catalyst-cooperative/pudl/pull/566
• fuel cost output to ref 860 generators. by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/574
• Ferc714 by @yashkumar1803 in https://github.com/catalyst-cooperative/pudl/pull/594
• Ei mcoe by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/592
• Transform function for distribution systems and other edits by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/643
• Add manually compiled balancing authority id fixes by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/646
• Transform function for AMI EIA861 by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/647
• Transform function for EIA 861 Dynamic Pricing Table by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/649
• Normalize the Balancing Authority Table and add a BA Association Table by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/651
• Transform func for Eia861 Green Pricing table by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/653
• Net metering table eia861 by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/671
• Service territories by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/670
• Non net metering function eia861 by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/680
• Categorize eia codes with either Util or BA priority by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/687
• Add a new FERC 714 Output Module by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/699
• 635: Datastore passes travis tests by @ptvirgo in https://github.com/catalyst-cooperative/pudl/pull/701
• Operational data table eia861 by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/691
• Add limit_by_state option to utility territory generation by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/707
• Simplify datapkg_to_sqlite script by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/712
• Clobber datapackage bundles not single datapackages by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/714
• Reliability and utility data eia861 by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/710
• Datastore improvements by @ptvirgo in https://github.com/catalyst-cooperative/pudl/pull/715
• Distributed generation eia861 by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/724
• Set up GitHub Actions to run Tox/PyTest by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/727
• Restore utility_assn() and other code wiped out by PR 724 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/730
• Energy efficiency eia861 by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/731
• Demand mapping by @yashkumar1803 in https://github.com/catalyst-cooperative/pudl/pull/717
• Ferc714 by @ptvirgo in https://github.com/catalyst-cooperative/pudl/pull/733
• Demand side management eia861 by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/732
• Some tweaks to table columns and data types by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/743
• get_census2010_gdf uses datastore by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/764
• Datastore data package validation and updated DOIs by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/761
• More robust flake8 linting by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/768
• Validate new dois by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/773
• Draft of ferc1 + eia860 + eia923 data integration for 2019 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/788
• Merge Sprint25 into dev branch by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/800
• Add DOIs for production archives on Zenodo by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/804
• Zipcode fix by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/820
• Better help messages and default to verbose logging by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/825
• Add docker build scripts by @rousik in https://github.com/catalyst-cooperative/pudl/pull/826
• Fix few issues surfaced in the previous PR by @rousik in https://github.com/catalyst-cooperative/pudl/pull/827
• Automate docker image builds by @rousik in https://github.com/catalyst-cooperative/pudl/pull/829
• Bump build-push-action to @v2 and fix arguments. by @rousik in https://github.com/catalyst-cooperative/pudl/pull/831
• Draft documentation framework for data sources by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/821
• Eia epa crosswalk by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/822
• Integrate EIA-860 2008 data by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/838
• Integrate EIA 860 M into ETL by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/824
• Add basic Datasette metadata and deployment script by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/841
• Notebook land: intro notebooks for CEMS and output tables by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/823
• add output and access notebooks by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/844
• Allocate generation_fuel_eia923 table data to generators by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/785
• Notebook land by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/853
• Ensure deterministic checksums on csv.gz outputs by @rousik in https://github.com/catalyst-cooperative/pudl/pull/856
• Add output methods for all remaining EIA 861 tables. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/862
• EIA860 old years (through 2004) by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/849
• Add high-performance timeseries anomaly detection and imputation module by @ezwelty in https://github.com/catalyst-cooperative/pudl/pull/871
• Speed up FERC 714 hourly demand transform by @ezwelty in https://github.com/catalyst-cooperative/pudl/pull/873
• Always run interim ETL tests b/c they're fast now. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/874
• Alaska is a thing by @rousik in https://github.com/catalyst-cooperative/pudl/pull/876
• Specify min/max versions for all dependencies in setup.py by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/875
• Fix broken links in README by @kyleries in https://github.com/catalyst-cooperative/pudl/pull/864
• Datastore refactoring by @rousik in https://github.com/catalyst-cooperative/pudl/pull/880
• Add unit test environment that runs quick tests under src/pudl by @rousik in https://github.com/catalyst-cooperative/pudl/pull/867
• Regex future warning by @rousik in https://github.com/catalyst-cooperative/pudl/pull/883
• Adjust FERC 714 service territories by using modified versions of EIA 861 tables by @ezwelty in https://github.com/catalyst-cooperative/pudl/pull/881
• Timeseries unittest by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/885
• Bugfixes for states=[ALL] and SQLite DB clobber check by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/890
• Consolidate interim ETL / output tests by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/892
• Jupyterhub beta by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/894
• Clean up PyTest config, coverage generation, unit tests by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/896
• Implementation of DataFrameCollection by @rousik in https://github.com/catalyst-cooperative/pudl/pull/887
• Sprint29 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/897
• Dev by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/898
• Pyarrow v3 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/912
• Eia860 validation by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/911
• Improvements to the DataFrameCollection by @rousik in https://github.com/catalyst-cooperative/pudl/pull/916
• pudl_datastore --list-partitions by @rousik in https://github.com/catalyst-cooperative/pudl/pull/925
• Pudl rmi by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/926
• Pytest scripts by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/913
• Sprint30 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/933
• Integrate EIA-860m through Nov. 2020 + fixed PUDL Plant IDs by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/934
• Update PUDL Development Docs by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/940
• Metadata docs by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/907
• Update transform documentation by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/939
• Convert Census DP1 to SQLite by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/948
• Sprint31 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/951
• Databeta by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/956
• Dev by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/957
• Dev docs setup updates by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/955
• Data Source Docs / Overview / Usage Modes by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/953
• Remove leading zeroes from all numeric EIA generator_ids. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/968
• Merge work from April 12-26 sprint. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/970
• Fix pudl ids by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1003
• Form instructions by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/987
• Simplified demand allocation algorithm by @yashkumar1803 in https://github.com/catalyst-cooperative/pudl/pull/741
• final review of release documentation for grammar, punctuation, spelling, and clarity issues by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1019
• Improve documentation and error messages related to use of EIA API keys. by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1030
• Convert spatial doctests to unit tests by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1021
• Merge accumulated dev changes from sprint ending 2021-07-05 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1068
• Integrate eia 923 2001-2008 by @TrentonBush in https://github.com/catalyst-cooperative/pudl/pull/1035
• Remove pytest logging capture to avoid duplicate logging output by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1077
• Integrate EPA CEMS 2020 data by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1078
• WIP: update allocation of net gen by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/989
• PUDL Unit ID Assignments by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1037
• update ferc1 documentation: by @aesharpe in https://github.com/catalyst-cooperative/pudl/pull/1053
• Release v0.4.0 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1087
• add xfail test for ownership tbl and make count_records FASTER by @cmgosnell in https://github.com/catalyst-cooperative/pudl/pull/1118
• Replace janky merge_on_date_year with built-in merge_asof() by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1103
• Release v0.4.0 by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1124
• Potential PUDL v0.4.0 release by @zaneselvans in https://github.com/catalyst-cooperative/pudl/pull/1125

New Contributors

• @rousik made their first contribution in https://github.com/catalyst-cooperative/pudl/pull/566
• @yashkumar1803 made their first contribution in https://github.com/catalyst-cooperative/pudl/pull/594
• @ptvirgo made their first contribution in https://github.com/catalyst-cooperative/pudl/pull/701
• @kyleries made their first contribution in https://github.com/catalyst-cooperative/pudl/pull/864

Full Changelog: https://github.com/catalyst-cooperative/pudl/compare/v0.3.2...v0.4.0

Energy Systems - Energy Data Accessibility and Integration - Python
Published by zaneselvans over 3 years ago

The Public Utility Data Liberation Project - v0.3.2: Integration of EIA 860 data for 2009-2010

The primary changes in this release:

• The 2009-2010 data for EIA 860 have been integrated, including updates
  to the data validation test cases.
• Output tables are more uniform and less restrictive in what they
  include, no longer requiring PUDL Plant & Utility IDs in some tables.
• This release was used to compile v1.1.0 of the PUDL Data Release, which is archived at Zenodo under this DOI: https://doi.org/10.5281/zenodo.3672068

With this release, the EIA 860 & 923 data now (finally!) cover the same span of time. We do not anticipate integrating any older EIA 860 or 923 data at this time.

Energy Systems - Energy Data Accessibility and Integration - Python
Published by zaneselvans about 5 years ago

The Public Utility Data Liberation Project -

Energy Systems - Energy Data Accessibility and Integration - Python
Published by zaneselvans about 5 years ago

The Public Utility Data Liberation Project -

Energy Systems - Energy Data Accessibility and Integration - Python
Published by zaneselvans about 5 years ago

The Public Utility Data Liberation Project -

Energy Systems - Energy Data Accessibility and Integration - Python
Published by zaneselvans over 5 years ago

The Public Utility Data Liberation Project -

Energy Systems - Energy Data Accessibility and Integration - Python
Published by zaneselvans over 5 years ago

The Public Utility Data Liberation Project -

Energy Systems - Energy Data Accessibility and Integration - Python
Published by zaneselvans over 5 years ago

The Public Utility Data Liberation Project -

Energy Systems - Energy Data Accessibility and Integration - Python
Published by zaneselvans over 5 years ago

The Public Utility Data Liberation Project -

Energy Systems - Energy Data Accessibility and Integration - Python
Published by zaneselvans over 5 years ago

The Public Utility Data Liberation Project -

Energy Systems - Energy Data Accessibility and Integration - Python
Published by zaneselvans over 5 years ago

The Public Utility Data Liberation Project -

Energy Systems - Energy Data Accessibility and Integration - Python
Published by zaneselvans over 5 years ago