Data

Tools

Indexes

Applications

Experiments

Open Sustainable Technology

A curated list of open technology projects to sustain a stable climate, energy supply, biodiversity and natural resources.

Recent Releases of Health Assessment Workspace Collaborative

Health Assessment Workspace Collaborative - 2024-Q4

January 28th, 2025

We are proud to release 2024.Q4. This release includes two large new features that are available to select users; after some internal testing we plan on making them generally available:

  1. Benchmark dose (BMD) modeling using pybmds. Nearly a decade ago HAWC was capable of BMD modeling using EPA's BMDS. The engineering was complex and required running modeling computation on a remote Windows computer. More recently BMDS has been rewritten to a python package pybmds that can be executed directly in HAWC. We've updated HAWC to use pybmds and with a new simplified modeling user interface. The interface is currently available for HAWC beta testers, but we hope to make available broadly in the next release
  2. New search capabilities. Searches have been available in HAWC since the beginning, but searches could only be conducted from within an assessment, that is, you could not search for content across assessments. With this release, you can search for studies and visualizations across assessments. When you're authenticated, you can search for content that may not be publicly released as well but still restricted to your level of access on projects.

In addition to these larger features, we've made some refinements to assessment labels for tables and visuals that were released last quarter, based on feedback that we've gotten from use.

Finally, we've made a large change to the header and global navigation pane of HAWC to better indicate to users they're logged in vs. logged out, as well as other minor UI updates based on feedback. We welcome feedback!

A total of 30 pull requests were merged from 2024-10-25 and 2025-01-28. The complete changelog is described in summary below; the emoji legend describes how change bullets are categorized:

Global Updates:

Labeling updates:

User Interface (UI) Updates:

Visualizations:

API Updates:

Developer/Security updates:

Emoji legend:

  • 🚀 Feature - this is a new feature or improvement to the existing application
  • 🧪 In Progress - a feature actively under development; may not yet be available in production
  • 🐞 Bug - fixed an error in the current software
  • 📄 Documentation - updated user/developer documentation
  • 🔒 Security Updates - security related updates
  • 🛠️ Developer Tools - updated internal tooling for a better developer experience

Natural Resources - Ecotoxicology - Python
Published by shapiromatron 3 months ago

Health Assessment Workspace Collaborative - 2024 - Q3

October 24th, 2024

We're proud to release 2024.Q3. This release features a new labelling tool, two new visualizations, and various export improvements.

Assessment members can now create and apply labels to visuals, data pivots, and summary tables to help categorize content within an assessment. Users can filter list pages by these labels and view all items with a given label on one page. This allows for better organization of various content within an assessment on HAWC. This is especially useful for large assessments with lots of visualizations in different health domains, for example.

Two new visualizations were introduced this quarter: PRISMA flow diagrams and literature Venn diagrams. PRISMA flow diagrams are available as a new visual type used for conveying an assessment's systematic review process. They were inspired by a recent paper by Walker et al. 2024. In the literature review module, users can compare tagged references using our new Venn diagram visualization option (alongside the existing tag tree visualization).

Export changes include a new style of reference export, a BMDS-friendly animal endpoint export, and exports for User Defined Fields. Existing imports have been improved, with new filters, new data fields, and drastic speed improvements for large animal exports.

A total of 51 pull requests were merged from 2024-07-10 to 2024-10-24. The complete changelog is described in summary below; the emoji legend describes how change bullets are categorized:

Global Updates:

  • 🚀 Assessment labels by @rabstejnek in #1085 (September 27)
  • 🚀 Study filter for exports by @munnsmunns in #1082 (August 28)
  • 🚀 Reparse DSSTox content and add search index by @munnsmunns in #938 (September 04)
  • 🚀 remove unused scripts by @shapiromatron in #1094 (September 05)
  • 🐞 Add form prefixes to HTMX forms by @caseyhans in #1100 (September 24)
  • 🐞 Catch FieldErrors and ValidationErrors in autocomplete by @munnsmunns in #1088 (August 28)

Study Evaluation / Risk of Bias:

  • 🐞 Fix for conflict between SET url setting and url validation by @rabstejnek in #1111 (September 26)

Assessment Updates:

  • 🚀 Update AssessmentValue global API by @shapiromatron in #955 (September 03)
  • 🚀 Make AssessmentValue.study M2M by @munnsmunns in #1093 (September 19)
  • 🚀 Remove Executive Summary feature by @munnsmunns in #1086 (August 29)
  • 🚀 ADAF updates for Assessment values by @munnsmunns in #1110 (October 01)

User Interface (UI) Updates:

  • 🚀 Update Downloads Page by @caseyhans in #1067 (September 03)
  • 🧪 Labeled Items List by @caseyhans in #1091 (September 16)

Literature Review:

  • 🚀 Long reference export by @shapiromatron in #1098 (September 18)
  • 🚀 Literature tag venn diagrams by @shapiromatron in #1092 (September 18)
  • 🐞 Bulk tagging JS fix by @rabstejnek in #1115 (October 03)

Animal Data Extraction:

  • 🚀 CPHEAPM-28 Add ToxRefDB to Update Modules by @ZindahFarhaICF in #1073 (September 10)
  • 🚀 CPHEAPM-26 - Add ToxRefDB vocab fixture and display by @ZindahFarhaICF in #1066 (September 09)
  • 🚀 CPHEAPM-27 Add ToxRefDB to Update Vocab Endpoint by @ZindahFarhaICF in #1075 (September 10)
  • 🚀 Animal export optimization by @rabstejnek in #1099 (September 20)
  • 🚀 Client Update + BMD Export by @caseyhans in #1079 (September 18)
  • 🚀 Add UDF to animal groups and datapivot exports by @caseyhans in #1104 (September 28)

Epidemiology Data Extraction:

  • 🚀 Add countries to epiv1 datapivot export by @caseyhans in #1107 (September 24)
  • 🚀 multiple worksheet epiv2 export by @shapiromatron in #1113 (October 03)

Visualizations:

  • 🚀 create prisma ui by @munnsmunns in #1072 (September 05)
  • 🚀 Add exports for Prisma plot by @munnsmunns in #1090 (September 05)
  • 🚀 Add exports to Prisma Config form by @munnsmunns in #1097 (September 09)
  • 🚀 Implement prisma diagram visual by @munnsmunns in #1105 (October 08)
  • 🚀 update epiv2 study design visual fields by @caseyhans in #1116 (October 04)
  • 🐞 Data pivot - fix conditional formatting on numeric data by @shapiromatron in #1122 (October 22)

Developer/Security updates:

  • 🛠️ Update Ruff Rules by @caseyhans in #1080 (August 28)
  • 🛠️ remove user_can_edit_assessment by @caseyhans in #1071 (August 28)
  • 🛠️ Add scientific notation filter by @munnsmunns in #1077 (August 28)
  • 🛠️ fix create_superuser command by @caseyhans in #1084 (August 28)
  • 🛠️ Alert tag rewrite by @caseyhans in #1074 (August 29)
  • 🧪 Add path for new HERO API in HERO-ID based import by @munnsmunns in #1070 (September 04)
  • 🛠️ make all hawc imports relative instead of absolute by @shapiromatron in #1095 (September 05)
  • 🛠️ Add coverage artifact by @shapiromatron in #1106 (September 18)
  • 🛠️ add a user agent for external requests by @shapiromatron in #1096 (September 18)
  • 🛠️ Change pandas pattern for empty dataframe by @caseyhans in #1109 (September 26)
  • 🛠️ Add minimum Python coverage number by @shapiromatron in #1120 (October 16)
  • 🐞 Fix display choices for swagger display /admin/api/swagger/ by @shapiromatron in #1121 (October 22)
  • 🔒 Bump webpack from 5.88.2 to 5.94.0 in /frontend in #1101 (September 10)
  • 🔒 update npm express by @shapiromatron in #1102 (September 10)
  • 🔒 Bump micromatch from 4.0.5 to 4.0.8 in /frontend in #1103 (September 10)
  • 🔒 Bump wagtail from 6.1.2 to 6.1.3 in #1068 (July 12)
  • 🔒 update django from 5.0.6 to 5.0.7 by @shapiromatron in #1069 (July 12)
  • 🔒 Upgrade django to 5.1 by @munnsmunns in #1083 (August 28)

Emoji legend:

  • 🚀 Feature - this is a new feature or improvement to the existing application
  • 🧪 In Progress - a feature actively under development; may not yet be available in production
  • 🐞 Bug - fixed an error in the current software
  • 📄 Documentation - updated user/developer documentation
  • 🔒 Security Updates - security related updates
  • 🛠️ Developer Tools - updated internal tooling for a better developer experience

Natural Resources - Ecotoxicology - Python
Published by caseyhans 6 months ago

Health Assessment Workspace Collaborative - 2024 - Q2

July 11, 2024

This release is a collection of smaller user-facing updates to improve new features we've added over the last year, and larger infrastructure changes required to maintain the overall healthiness of the project.

For user-facing updates, we've continued to expand the functionality of User Defined Forms (UDF) released last quarter, adding a reference UDF display, data export downloads, and fixing a few bugs identified during testing. We've made some minor visualization updates as well - adding study evaluation data to epiv2 data pivot, and allowing users to determine which column to use to generate counts in exploratory heatmaps.

Our infrastructure for generating data exports was overhauled for speed and code readability; this was an 8-month effort and thousands of manual visualization inspections to ensure backwards compatibility with our previous approach. We've also expanded our test coverage to 92%, writing dozens of new tests to improve the stability and quality guarantees of HAWC.

A total of 50 updates were made from 2024-04-16 to 2024-07-11. The complete changelog is described in summary below; see the emoji legend which describes how changes are categorized:

Global Changes:

  • 🚀 Allow anonymous contact us w/ turnstile enabled by @shapiromatron in #1018 (April 19)
  • 🚀 Add a SQL select statement by @shapiromatron in #1035 (June 05)
  • 🚀 Email verify post by @shapiromatron in #1019 (June 17)
  • 🚀 Rewrite exports mechanism for large datasets (exports v2) by @rabstejnek in #1054 (July 02)
  • 🐞 Update (exports v2) from image review by @shapiromatron in #1058 (July 02)

Assessment Updates:

  • 🚀 Assessment Detail/Value adjustments by @munnsmunns in #1028 (June 05)

Literature Screening:

  • 🐞 Fix identifier DOI dupe checks by @caseyhans in #1038 (June 05)
  • 🐞 Fix bugs from demo by @caseyhans in #1061 (June 28)
  • 🐞 UDF & Reference Filterset Change by @caseyhans in #1064 (July 02)

Data Tables and Visualizations:

  • 🚀 Add RoB data to EpiV2 exports by @munnsmunns in #1039 (June 13)
  • 🚀 Exploratory heatmap "count column" setting by @rabstejnek in #1034 (June 14)
  • 🧪 Add PRISMA visual type view and JS stub by @munnsmunns in #1020 (June 13)
  • 🐞 Fix study evaluation sort_order by @shapiromatron in #1052 (July 02)
  • 🐞 SEM scroll fix by @caseyhans in #1013 (June 06)

User Defined Fields:

  • 🚀 Standardize 'User Defined Fields' for UDFs by @munnsmunns in #1022 (June 06)
  • 🚀 Display Reference UDF Data by @caseyhans in #1026 (June 14)
  • 🚀 Add UDF logs by @caseyhans in #1023 (June 17)
  • 🚀 Add UDF Export API by @caseyhans in #1027 (June 20)
  • 🐞 UDF & Reference Filterset Change by @caseyhans in #1064 (July 02)

Ecological Data Extraction:

  • 🚀 Eco - updates from review by @caseyhans in #1056 (June 28)
  • 🚀 Eco - add deprecated badge on search view by @shapiromatron in #1060 (June 28)

Bugfixes:

  • 🐞 Bugfixes from 2024.Q1 release by @shapiromatron in #1015 (April 19)
  • 🐞 Make EHV download public by @caseyhans in #1036 (June 05)
  • 🐞 Fix profile update redirect by @caseyhans in #1037 (June 05)
  • 🐞 Clone name and ensure under size limit by @shapiromatron in #1021 (June 15)

Developer/Security updates:

  • 🛠️ New permission classes by @shapiromatron in #1016 (April 19)
  • 🛠️ Remove react-datepicker by @munnsmunns in #1017 (May 13)
  • 🛠️ Update to PostgreSQL v 16 by @shapiromatron in #1029 (May 14)
  • 🛠️ Remove uswds by @shapiromatron in #1031 (May 14)
  • 🛠️ Remove requirements.txt by @shapiromatron in #1033 (June 05)
  • 🛠️ Refactor HTMX Viewsets by @shapiromatron in #1012 (June 14)
  • 🛠️ Developer quality of life improvements by @shapiromatron in #1040 (June 15)
  • 🛠️ python package updates by @shapiromatron in #1046 (June 17)
  • 🛠️ Replace and with bs4 tags by @munnsmunns in #1049 (June 27)
  • 🛠️ Improve test coverage by @munnsmunns in #1051 (June 27)
  • 🛠️ Add dev-header to full width of body instead of as a tab by @shapiromatron in #1057 (June 28)
  • 🛠️ Add form helpers instead of repeating code by @shapiromatron in #1053 (June 28)
  • 🛠️ Remove supported browser check - no longer relevant by @shapiromatron in #1059 (July 01)
  • 🛠️ Add more tests by @shapiromatron in #1062 (July 01)
  • 🔒 Update gunicorn by @shapiromatron (April 24)
  • 🔒 Bump @babel/traverse from 7.22.8 to 7.24.5 in /frontend in #1025 (May 14)
  • 🔒 Bump webpack-dev-middleware from 6.1.1 to 6.1.2 in /frontend in #1030 (May 14)
  • 🔒 Bump glob-parent from 3.1.0 to 5.1.2 in /frontend in #1032 (May 14)
  • 🔒 Update github action versions by @shapiromatron in #1024 (May 15)
  • 🔒 Bump jinja2 from 3.1.2 to 3.1.4 in #1042 (June 05)
  • 🔒 Bump requests from 2.31.0 to 2.32.2 in #1041 (June 05)
  • 🔒 Bump express from 4.18.2 to 4.19.2 in /frontend in #1043 (June 15)
  • 🔒 Bump postcss from 8.4.27 to 8.4.38 in /frontend in #1044 (June 15)
  • 🔒 Bump braces from 3.0.2 to 3.0.3 in /frontend in #1045 (June 15)
  • 🔒 Bump es5-ext from 0.10.62 to 0.10.64 in /frontend in #1047 (June 17)
  • 🔒 Bump djangorestframework from 3.15.1 to 3.15.2 in #1063 (June 28)
  • 🔒 Switch from python:3.12-slim-bullseye to centos:stream9-minimal by @shapiromatron in #1065 (July 02)

Emoji legend:

  • 🚀 Feature - this is a new feature or improvement to the existing application
  • 🧪 In Progress - a feature actively under development; may not yet be available in production
  • 🐞 Bug - fixed an error in the current software
  • 📄 Documentation - updated user/developer documentation
  • 🔒 Security Updates - security related updates
  • 🛠️ Developer Tools - updated internal tooling for a better developer experience

Natural Resources - Ecotoxicology - Python
Published by shapiromatron 10 months ago

Health Assessment Workspace Collaborative - 2024 - Q1

April 15, 2024

This release includes a new feature we've been developing for the last six months or so - User Defined Forms (UDF). HAWC has always had a fixed schema for data extraction; this was an intentional decision because we wanted to be able to visualize content and perform standardized data analysis across assessments. However, it came at the expense of flexibility. Adding UDFs will allow users to augment existing forms with custom fields for their assessment. Right now, we've added UDFs to literature tags, studies, and bioassay endpoints, but we expect to add more UDF options to other items in the future. We plan on adding UDF data exports and integration with existing visuals in the next release. Documentation is limited, but please reach out with any questions or comments - we'd love to hear your feedback!

Other notable updates include new options in Data Pivot visualizations (calculated columns, columns to the right of the plot), updated task table filtering, updated assessment analytics for epidemiology and study evaluation domains, and many minor user-interface improvements throughout HAWC based on user feedback.

A total of 42 pull requests were merged into HAWC from 2024-02-02 to 2024-04-15. The complete changelog is described in summary below; see the emoji legend which describes how changes are categorized:

Global Changes:

Assessment:

Literature Screening and Tagging:

Data Extraction:

Visualizations:

Developer/Security updates:

Emoji legend:

  • 🚀 Feature - this is a new feature or improvement to the existing application
  • 🧪 In Progress - a feature actively under development; may not yet be available in production
  • 🐞 Bug - fixed an error in the current software
  • 📄 Documentation - updated user/developer documentation
  • 🔒 Security Updates - security related updates
  • 🛠️ Developer Tools - updated internal tooling for a better developer experience

Natural Resources - Ecotoxicology - Python
Published by munnsmunns about 1 year ago

Health Assessment Workspace Collaborative - 2023 - Q4

February 1, 2024

We're proud to release 2023.Q4. This release makes a number of improvements to our literature screening workflow, an assessment dashboard to highlight key analytics for your project, and a long-needed refresh for improving flexibility and performance of our data exports used in visualizations. Finally, we added a new visual type - an image upload, so we can more easily allow for other visualization types for which HAWC may not be capable of creating.

In the last six months we've released a number of updates in the literature screening module, and we've had some real-world applications which helped identify a few improvements to make the module more intuitive. The first is a workflow system, which allows users to set rules to create workflows for literature (for example, a title/abstract review, a full-text review, etc.). In addition, we added a bulk merge feature to conflict resolutions - this should make it easier to rapidly resolve conflicts, or accept all tags even if only one user has reviewed them (which can be helpful in some situations). We hope you'll find these features useful!

We have a few features that we're close to releasing that we made great progress on with this release. The first is an internal documentation site, so we can better add content for how to use HAWC and keep it up to date for our user base. The second is dramatic performance improvements to our exports, which should make them more flexible, and easier to add to new parts of our app (along with new visualization types). The third are user defined fields, which should be added in the future to our literature screening workflow as well as key data extraction models. We hope that these features will be available in our next release.

A total of 50 pull requests were merged into HAWC from 2023-10-07 to 2024-01-18. The complete changelog is described in summary below; see the emoji legend which describes how changes are categorized:

  • Global Updates:

    • 🚀 Allow Sharepoint (sharepoint.com), Zenodo (zenodo.org), and Canadian Government (canada.ca) hyperlinks #924 #949
    • 🚀 Add new option for users to show HAWC in debug mode #951
    • 📄 Add a documentation application to add internal documentation for HAWC editors #950
    • 🛠️ Update EPA flavor of application to use 2023 style guide #945 #960

  • Assessment:

    • 🚀 Create new assessment-level analytics pages #917 #927
    • 🚀 Add study-level (one row per study) data summaries to downloads page #944
    • 🚀 Add new options for downloading published/unpublished data to assessment downloads #956
    • 🐞 Fix assessment lists from showing duplicates #930

  • Literature Review:

    • 🚀 Add literature screening workflows #952
    • 🚀 Add bulk merge tag conflicts, and more filters for conflict resolution #919 #932
    • 🚀 Add additional duplicate reference check on imports #965
    • 🚀 Improve layout for references #933 #953

  • Study Evaluation / Risk of Bias:

    • 🐞 Restore user study evaluation review dashboard #964
    • 🐞 Fix bug in creating scores via API #936
    • 🛠️ Improve speed and standardize all study evaluation APIs #934
    • 🧪 Rewrite export engine for improved performance #921

  • Data Extraction: Animal Bioassay:

    • 🚀 Update animal dose response plots #780
    • 🚀 add Effect Tag creation (Bioassay, Epi, etc.) to API #937

  • Data Extraction: Epidemiology:

    • 🚀 Add client for modifying epi v2 data #888
    • 🧪 Rewrite exports for improved performance #911 #922

  • Data Extraction: In Vitro:

    • 🧪 Rewrite exports for improved performance #958

  • Visualizations:

    • 🚀 Add new visual type - user uploaded image #941
    • 🚀 Overhaul data prefilters and add new prefilter options #939 #962
    • 🚀 Automate downloading visuals using the client #810
    • 🐞 Use preferred assessment naming convention for risk of bias/study evaluation in visuals #946

  • Developer/Security updates:

    • 🔒 Update to Python 3.12 #967
    • 🔒 Update to Django 5 #959
    • 🔒 Update python packages to most recent versions #928 #929
    • 🔒 Update pydantic to v2 #948
    • 🛠️ Improve view/form test code coverage #905
    • 🛠️ Remove/refactor unused and duplicative code #799 #925 #926 #931
    • 🛠️ Standardize template tags for easier use #920 #923
    • 🛠️ Standardize assessment level APIs #935
    • 🛠️ Standardize caching patterns #940
    • 🛠️ Replace black with ruff formatter #942
    • 🐞 Remove broken Zenodo badge from readme #954
    • 🐞 Fix flaky tests to improve code stability #943 #968

Emoji legend:

  • 🚀 Feature - this is a new feature or improvement to the existing application
  • 🧪 In Progress - a feature actively under development; may not yet be available in production
  • 🐞 Bug - fixed an error in the current software
  • 📄 Documentation - updated user/developer documentation
  • 🔒 Security Updates - security related updates
  • 🛠️ Developer Tools - updated internal tooling for a better developer experience

Natural Resources - Ecotoxicology - Python
Published by shapiromatron about 1 year ago

Health Assessment Workspace Collaborative - 2023 - Q3

October 17th, 2023

We're proud to release 2023.Q3! We've made a number of quality-of-life improvements this quarter based on user feedback. First, we've refined searching and filtering during literature screening and have updated how references are displayed. Second, we've added a new data export for study evaluation heatmaps; this export is now available on all existing visuals. Finally, we've added some new features to data pivots and exploratory heatmaps to enable more customization of their look and feel.

There's more work that we've been doing behind the scenes that we hope will allow us to add some substantial new features in future release. First, we've rewritten how data can be filtered for visualizations so we can filter more quickly and flexibly. This should allow us to add more visuals and filtering options in the future. Second, we're adding User Defined Forms (UDF), so that users can augment the existing tagging and extractions in HAWC with additional information that may be required for an assessment. This has long been requested over the years and we're excited to break ground on this work.

The complete changelog is described in summary below; see the emoji legend which describes how changes are categorized:

  • User Interface (UI) Updates:

    • 🚀 Added Unpublished badge to studies, visuals, etc to better show team members the status of content under development #860
    • 🚀 Text areas can be resized globally on all text boxes #883
    • 🧪 Download filtered data for outcome lists (animal, epidemiology, etc.) #895
    • 🐞 Converted assessment Log search form to new inline filter form layout #872

  • Literature Review:

    • 🚀 Added authors and year filter to literature review reference search #886
    • 🚀 Add better full-text search capabilities #877
    • 🚀 UI updates for References - simplify layout for detail and list reference views #878
    • 🚀 Added a new "accept all" to accept all proposed tags during conflict resolution #908
    • 🚀 Improve validation for full text URL uploads and display errors to users #887
    • 🐞 Add db restraint to prevent duplicate (reference, user) pairs in User tags #890
    • 🐞 Fix bug in keyword highlighting for literature tagging with special characters #878

  • Study Evaluation / Risk of Bias:

    • 🐞 Fix duplicative studies shown in ROB assignment views #885

  • Data Extraction: Animal:

    • 🚀 Update experiment comments label on detail view to be consistent with form #904
    • 🚀 Animal summary endpoint exports now contain sample size 'N' #898
    • 🚀 Calculate percent change relative to control calculated for dichotomous when possible #909 #915
    • 🐞 Fixed bug where users couldn't set or view NOEL/LOEL for endpoints where data was not extracted #907

  • Data Extraction: Epidemiology:

    • 🚀 Added adverse direction field to epidemiological study population Data Extractions #902
    • 🚀 Rewrite export view for performance #870
    • 🚀 Update API to filter study designs by study identifier #918

  • Visualizations:

    • 🚀 Study Evaluation Heatmaps - download data in tabular format using the actions button #892
    • 🚀 Exploratory Heatmap - toggle display of counts and colors #916
    • 🚀 Data Pivot - show arrows for confidence intervals when out of domain #874
    • 🚀 Data Pivot - allow more flexibility with background colors and gridlines #874
    • 🚀 Data Pivot - add more flexibility and precision with setting x-axis domains and display #874
    • 🚀 Data Pivot - full read/write control via API #896
    • 🚀 Visuals - full read/write control via API #896
    • 🐞 Data pivot file upload error #894
    • 🐞 Fix typos in visual list text #893
    • 🐞 Fix deprecated Endpoint Aggregation visual form and remove dependencies #883

  • User Defined Forms (UDF):

    • 🧪 Allow users to define their own forms #891 #882 #897
    • 🧪 Bind UDF to data extraction components in an assessment #899
    • 🧪 Bind UDF to tags in an assessment #899

  • Developer/Security updates:

    • 🛠️ Installed django-browser-reload package for automated browser reloading in debug #881
    • 🛠️ Remove dead code #900
    • 🛠️ Convert text fields to JSON fields for visuals and data pivots #901
    • 🛠️ Refactor prefilters for visuals and datapivots #876
    • 🛠️ Standardize actions button menus #860
    • 🛠️ Added utility method for generating tabular flat exports to remove duplicate code #873
    • 🔒 Add setting to disable login attempts on some domains #879
    • 🔒 Fixes from security scans #875 #884 #889
    • 🔒 Updated Django to latest release #903
    • 🔒 Updated docker container versions #910

Emoji legend:

  • 🚀 Feature - this is a new feature or improvement to the existing application
  • 🧪 In Progress - a feature actively under development; may not yet be available in production
  • 🐞 Bug - fixed an error in the current software
  • 📄 Documentation - updated user/developer documentation
  • 🔒 Security Updates - security related updates
  • 🛠️ Developer Tools - updated internal tooling for a better developer experience

Natural Resources - Ecotoxicology - Python
Published by dannypeterson over 1 year ago

Health Assessment Workspace Collaborative - 2023-Q2

August 9th, 2023

We're proud to release 2023.Q2. Over the last few quarters we've been internally developing new data extraction and visualization components in partnership with our scientists, and with this release, those features are made generally available. Specifically:

  • Last year we released new data extraction modules for Epidemiology. This update integrates data extraction with existing with data visualization in HAWC, such as exploratory heatmaps and data pivots.
  • We've released ecological data extraction, which, like epidemiology, is now fully integrated with our exploratory heatmap and data pivot visualizations.
  • We've added an assessment metadata module to add high level project data to assessments (assessment type, current status, related documentation), as well as adding assessment values, if a HAWC assessment results in quantitative values as a conclusions.

In addition to these changes, we've overhauled searching and filtering throughout HAWC; filters should be faster with intuitive. In addition, we added new customizations to assessments to show/hide items on the sidebar, to make it easier to only have links throughout an assessment to pages which have data.

The complete changelog is described below (features marked 🧪 indicate they're not yet generally available and are behind feature flags):

New Features:

  • General improvements
    • Redesigned search/filters to include an inline search bar and expandable 'advanced search' filter form #817 #838
    • Improve styling for Excel-based tabular exports #816
    • 🧪 New APIs for searching assessments by chemical and reference (admin only) #826
  • Literature screening
    • Add wildcard keyword searching for literature screening #833
    • Remove ability to change source on already executed literature import #849
    • Steer users to importing references from PubMed/HERO instead of manual extraction #819
  • Assessment configuration
    • Assessment sidebar customization allows users to show/hide all sidebar links #827
    • Restyled Assessment Downloads page which includes multiple different export types #807
    • Improve flexibility for searching and filtering assessments #839 #857
  • Study Evaluation
    • Add new study evaluation judgment choice: Minor/Critical concerns (sensitivity) #848
    • Improve display of study evaluation criteria when empty #821
  • Data Extraction: Animal Bioassay
    • Update export for bioassay data to include diagnostics field #850
    • Added Pubertal and Juvenile to animal constants #864
  • Data Extraction: Epidemiology
    • Added the ability to create data pivots and exploratory heatmaps using extracted data #832 #869
    • Add REST-API for epiv2 data extraction (create, read, update, delete) #820 #866
    • Added bulk-cleanup features to clean manually extracted data #855
  • Data Extraction: Ecology
    • Added ecological data pivots and exploratory heatmap visualizations using extracted data #845
    • Make ecological data extraction generally available #868
    • Added schema to developer and API documentation #841 #866
  • Assessment Values and Details
    • Additional revisions to assessment values and details per scientist feedback #792 #831
    • New API for CRUD operations of assessment values and details #858 #862
    • Make values and details generally available #868
  • Summary Tables and Visualizations
    • Streamline exploratory heatmap user interface #846
    • Update Evidence Profile Table language per scientist recommendation #837
    • 🧪 New visualization types using plotly (admin only) #804
  • Assorted User Requests
    • Assorted UI improvements #818 #823 #824 #822 #863
    • Updated notifications to streamline interactions with pages #819 #821 #844 #843 #855
    • Rebuild mgmt views to use templates instead of JS #808
    • Update CCTE Dashboard API to newly release version for chemical information #828

Software upgrades/security/stability:

  • Updated to Python 3.11 and assorted python packages #769 #847
  • Update Django from 3.2 to 4.2 LTS #806
  • Removed Blog, topic models, MyST-parser, and R client #851 #852 #853
  • Remove FIPS mode monkeypatch; fixed with upgrade to Django 4.2 #825
  • Fixed bug in migration dependency orders #842
  • Fixed bug in epiv2 data export failing with no adjustment factors #840
  • Fixed bug when searching bioassay results by invalid dose units #856
  • Fixed bug in bulk tagging when an Excel file has boolean tag fields #829
  • Fixed bug in user-interface when a permission error looked like a loading error #813
  • Fixed bugs in view logic resulting in page rendering failure #811 #812
  • Rewrite how large exports are generated using ORM more effectively #865
  • Handle exceptions triggered by security scans #871

Developer experience:

  • Updated black, replaced flake + isort with ruff #778
  • Update build tool to flit; migrate to pyproject.toml #854
  • Created CI/CD coverage report with all pull requests #859
  • Rename Views and ViewSets for clarity and consistency #814 #815
  • Rewrite DataPivotExtension to more generic Interactivity module #861
  • Update yarn v1 to v3, and then rollback #806 #867

Natural Resources - Ecotoxicology - Python
Published by dannypeterson over 1 year ago

Health Assessment Workspace Collaborative - 2023-Q1

April 13th, 2023

We're proud to release 2023.Q1. The primary feature in this release is the updated literature screening module with optional conflict resolution of tags. This features allows dual-screening of references in HAWC, and we hope this expands the types of systematic review methodologies that can be conducted from within HAWC.

In addition, we've made assorted quality of life updates throughout the application - assessment data downloads should be better organized and with more information about the underlying reference, study evaluation metrics and domains are now organized by health domain, and summary tables can now be set to filter and paginate for longer tables.

Finally, we're enabling a feature-flag system for future development to make it easier to add features that we can progressively roll out to users. We hope that this approach makes it easier for developers to update code with less context-switching, which ultimately should make us more efficient.

The complete changelog is described below (features marked 🧪 indicate they're not yet generally available and are behind feature flags):

New Features:

  • Literature Screening #756 #774 #752 #794 #789

    • release literature tagging with conflict resolution
    • rewrote literature summary pages and add new summary stats
    • added reference counts to all pages where tag lists are displayed
    • add user tag exports and filterable web views

  • Study Evaluation/Risk of Bias

    • update the study evaluation domain/metric view to show questions by study type #790
    • update study evaluation questions from NTP/OHAT and EPA/IRIS to current approaches; created tools to automate syncing in the future #771
    • add study evaluation metric key field for easier copying of metrics across assessments #771

  • Data Extraction

    • add API endpoint for epiv2 metadata settings, useful for automated machine extractions #766
    • updated animal data extraction help text #764
    • 🧪 release candidate for ecological data extraction #740 #746 #783 #784 #791
      • revised browsing of controlled vocabulary; added excel-based exports
      • add data export for extracted data
    • 🧪 release updated version of BMDS execution using 2022 version #788

  • Summary tables

    • add searchable, filterable, paginated generic summary tables #786
    • add a caption to summary tables and their corresponding exports #772
    • swap (again) columns in EPT tables #773

  • Summary visualizations

    • No changes this quarter

  • General Improvements

    • simplify assessment downloads page to only show exports with data #784
    • add reference IDs (PubMed, HERO, DOI) to common assessment download exports #745
    • 🧪 updates for assessment values #759
      • added admin API to retrieve a report of all values across assessments

  • Authentication/authorization

  • Client updates #765 #803

    • release version 2023.2
    • Added client.riskofbias.compare_metrics - align metrics when copying across assessments via fuzzy text matching
    • Added client.summary.datapivot_list - retrieve all data pivots for an assessment
    • Added client.summary.table_list - retrieve all tables for an assessment
    • Added login parameter to client.summary.set_authentication_token, to setup a client which can be used to browse the HAWC website
    • Added client.lit.reference_user_tags - retrieves all user tag for all references for an assessment.

  • Administrative

    • (optionally) add sentry #768
    • added assessment admin notes, to capture details on individual assessments #763

Bug/regressions fixed:

  • fix bug in data pivot initial settings #770
  • fix bug in hawc client and API sessions #776
  • fix bug in references "manually added" layout in html #779
  • fix bug in user ordering for study evaluation pages #781
  • remove duplicate study evaluation rows #782

Software upgrades/security/stability:

  • update python, javascript, and CSS dependencies #787 #777
  • remove old unused code #767
  • remove redundant views, old javascript patterns, and catch errors with invalid data inputs #803

Developer experience:

  • simplify django permissions for views and APIs #732 #801 #803
    • the old approach was confusing and difficult to infer or test
  • make browsable API available for PandasRenderers API views #800
    • this will be useful to debug query counts in django debug toolbar
  • added plotly template tag for easier plotting on django templates #793
  • added feature flags for features still undergoing internal review #802 #788
    • this makes it easier for develop to develop this features, but means we'll need to make sure code that we don't yet available to standard deployments is appropriately captured behind feature flags
    • in staging environments, we will scan with all features enabled, to ensure that all possible routes are undergoing security review

Natural Resources - Ecotoxicology - Python
Published by shapiromatron about 2 years ago

Health Assessment Workspace Collaborative - 2022-Q4

January 5th, 2023

The bulk of development this quarter was focused on new conflict resolution workflows for literature screening. If activated for an assessment, multiple users are required to select tags to a reference before they are applied. If two users tag a reference identically, the tags are marked as "resolved" and the reference advances to the next stage of screening. If two users tag a reference differently, then the tags are marked as conflicting, and the reference is added to the conflict resolution view. Note that existing assessments will not have this feature enabled, and tagging will work as it has in the past. This work is still not yet live, and is undergoing testing by our users (indicated by 🧪).

Another large-scale update was migrating our search/filter capabilities throughout HAWC to use django-filters instead of selectable, which is no longer maintained. Performance should be greatly improved with this migration, resulting in faster queries and page loads.

The complete changelog is described below:

New Features:

  • General Improvements

    • Update list views to use django-filters #704 #719 #736 #742 #729 #760

  • Literature Screening

    • Add PubMed and DOI identifiers on HERO import #755
    • Handle and report 500 errors from HERO requests #705
    • Hide RIS import button in some HAWC deployments #705
    • 🧪 Conflict resolution to our literature screening/tagging workflow #739
      • Create wireframes, new models, new view stubs, and new backend logic as a foundation for further conflict resolution development #708 #721 #716 #735 #741 #750
      • Add new view for users to identify and resolve tag conflicts #734 #749
      • Reorganize screening page layout to account for user feedback and new conflict resolution #714, #731, #738, #747

  • Risk of Bias/Study Evaluation

    • Update API to allow project managers to copy study evaluations across assessments #706
    • Make cloning risk of bias data across assessments easier using the HAWC client #724

  • Data Extraction

    • Bioassay
      • prepare for updating BMDS in HAWC; disable execution of current versions #711
    • Epidemiology (v1)
      • Result group names now include comparison set name on data entry forms #748

  • Summary tables

    • Moved factor column locations in EPTs #744
    • Add client API method to fetch all summary tables #717

  • Summary visualizations

    • Add Tableau data filters and modernize API #758 #761
    • Enable JPEG exports of visualizations #712
    • Improve to tagtree hover/help text #709
    • Only show units available in an assessment when configuring a data pivot #709
    • Fix data pivot plot layout to always show lower-bound on plot instead of on axis #710
    • Add client API method to fetch all summary visualizations #717

  • Authentication/authorization

    • Add deployment feature flag to prevent account creation by users #718
    • Make session cookies configurable #754
    • Add token-based authentication to access django views #762

  • Administrative

    • Update docs to show latest database schema #720

Bug/regressions fixed:

  • Fixed an edge case where dose-group counts in data pivots did not add up correctly #713
  • Fix case where user could unintentionally delete tagged references #753
  • Change test to use regex for content type ID stability #725
  • Various style fixes #726

Software upgrades/security/stability:

  • Replace type hints from typing module with built in object types #723
  • Standardize buttons for easier integration testing #743
  • Add eslint plugin for sorting imports #728
  • Reduce scope of API responses in production #715
  • Update github actions #730
  • Rename urls that use /new/ to use /create/ for consistency across HAWC #707
  • Remove dead code #722 #727

Natural Resources - Ecotoxicology - Python
Published by casey1173 over 2 years ago

Health Assessment Workspace Collaborative - 2022-Q3

October 6, 2022

Our major update this quarter includes deployment of a new extraction workflow and schema for epidemiology data. Named epiv2 in the software; all new assessments and existing assessments without epidemiology data extraction have been migrated to the new schema; assessments with the old schema, epiv1, are unchanged. The updated schema simplifies the old schema and user interface for data extraction and makes it easier to extract data at a high level (for systematic evidence maps, SEM), as well as a more detailed granular level, depending on the nature of the assessment. This release includes all the required components for data extraction; future releases will include the components needed to build summary tables and visualizations - in the meantime, try out the extraction forms and let us know what you think!

In addition, we made the following larger changes:

  • We updated assessment audit-logs support to create new exports and improving our current logs to show more information
  • We updated and standardized many of the forms that we use in HAWC. We've rewritten the text input fields, as well as the autocomplete/autosuggest fields using more modern, stable, and secure technologies. We hope this makes for a more seamless experience for users
  • We enhanced summary visuals and tables to reflect the needs of users integrating the visuals into their projects - we hope these updates make the current visuals more flexible

Two other major changes have been released in our next branch this quarter. These features are available under deployment feature flags and are undergoing some testing by users prior to merging into our main release branch; they are indicated below by the 🧪 emoji. Some production deployments may not have these features available; we hope to have them live soon:

  • 🧪 Capturing additional assessment level metadata on a HAWC assessment and quantitative values for a final conclusion. In larger organizations, HAWC is frequently one of many tools and resources used for conducting a systematic review and health assessment. Therefore, we've added some new high-level assessment level fields to capture information regarding the larger project (class of project, relevant documents or URLs, QA procedure and methodology), and fields for capturing health values of interest (points of departure, reference values, etc). #674
  • 🧪 Capturing ecological data of interest in a health assessment. Some health assessments include impacts on the environment and ecological health as well as human health. We've created a new data extraction module to extract fields in this domain as well, when appropriate. #634

The complete changelog is described below:

New features:

  • Overall Assessment Settings

    • Added downloads to assessment audit logs; expanded log views show additional details #683
    • 🧪 Created new fields for assessment-level metadata and assessment-level values #674

  • Literature Screening

    • Standardize DOIs in HAWC (lowercase; improved for deduplication) #666
    • Expand RIS import validation to catch more errors prior to import #647
    • Added additional warnings to prevent accidental search deletions #662
    • Update reference ID export to include references with no external IDs #659

  • Data extraction

    • Study
      • User-interface updates for high-level study information #656 #663
    • Animal
      • Added treatment related effect to animal endpoint groups #681
    • Epidemiology
      • Finalized an updated epiv2 data extraction module; basic Excel exports #642 #673 #687
      • Added "Ecological" and "Other" as choices for epiv1 study design; consistent with epiv2 #702
    • Ecology
      • 🧪 Added forms #634

  • Risk of Bias/Study evaluation

    • Expand filters for risk of bias assignments to include user #665
    • Updated default study evaluation metrics and domains #682

  • Summary tables and visualizations

    • Literature tagtrees
      • Added a legend #684
      • Added an option to show or hide reference counts, hiding counts is useful for documenting the approach in the planning and protocol phase #684
      • Added the ability to drag and drop nodes for complex tree layouts #684
      • Fixed bug where reference lists for a node were improperly filtered #654
    • Data Pivots
      • Updated forest plots to include treatment related effects in addition to statistical significance; can be used for conditional formatting #681
      • Updated barcharts to handle standard deviation for error bars (previously only confidence intervals were handled correctly) #651 #697
      • Added numeric duration exposure, useful for sorting #651
    • Study Evaluation Heatmaps/Barcharts
      • Improved default positioning of legend #685
    • Evidence Profile Tables
      • Improvements to evidence profile tables and reporting #675 #657
    • Study Evaluation Tables
      • Update Study Evaluation Table (SET) data entry based for easier data entry #678 #653

  • General improvements

    • Updated the EPA header and footer to match the new style guidelines #693 #703
    • Fix mobile rendering of website (sidebar collapse) #672 #700 #664
    • Modernized HTML text data entry fields to improve consistency and stability #655 #699
    • Modernized text autocomplete data entry fields to improve consistency and stability #667 #698 #701

  • Administrative

    • Updates to diagnostic API endpoints #668 #669

Bug/regressions fixed:

  • Fixed bug in downloading SVG exports of visuals #677
  • Fixed bug which prevented updating assessment file attachments #695
  • Better handle improper Unicode in Excel exports #652 #670
  • Report an error on extremely large exploratory heatmaps instead of crashing the browser #671
  • Improve validation error messages for users #658 #643
  • Fix assorted errors #646 #696 #679 #676 #692

Software upgrades/security/stability:

  • Quality of life improvements to developer experience #689 #660
  • Rewrote integration tests using Playwright instead of Selenium #650
  • Rewrote documentation using more user-friendly markdown instead of sphinx #680
  • Fix deployment on servers with FIPS cryptography enforcement (RHEL8) #688
  • Update HAWC configuration to reasonable and secure defaults #649 #691

Natural Resources - Ecotoxicology - Python
Published by rabstejnek over 2 years ago

Health Assessment Workspace Collaborative - 2022-Q2

July 12, 2022

Updates were light this quarter; we were focused on other projects. One major change is rewriting how rasterized visuals are created; we're using native tooling in the browser now instead of generating server side; this should result in increased performance and accuracy to the rendered visual. #630

The complete changelog is described below:

  • Visualizations:
    • Client side rasterization of SVG images #630
  • Literature screening:
    • Fixed literature tagtree overflow #638
    • Delete untagged references if search scope decreases #639
  • Study evaluations:
    • Add a "Save and continue editing" button for study evaluations #645
    • Standardize "completion" on frontend/backend for study evaluations #648
  • Client:
    • Update hawc client (version 2022.7) with new commands #573
  • Admin:
    • New admin dashboard - changes per day #636
    • New configurable hawc admin root #637
    • Improved admin performance #638
  • Developer updates:
    • Database commands for windows environments #640
    • Standardize fake data names #641
  • Current in development (behind feature flags):
    • ecology data extractions #620
    • epidemiology version 2 updates #632

Natural Resources - Ecotoxicology - Python
Published by shapiromatron almost 3 years ago

Health Assessment Workspace Collaborative - 2022-Q1

April 12, 2022

Our major feature this quarter is the ability to create Study Evaluation Tables, or SETs. These tables are designed to present study evaluations (or risk of bias) findings for multiple studies, but are more customizable than the existing heatmap tables we build. Users can select which studies and metrics to present, can create custom columns, and can even show the same metric across multiple health outcomes. For animal bioassay data, you can include additional metadata for a study such as study design, doses and dose units, or endpoints extracted.

In addition to the SET tables, we've made a few other notable changes:

  • After hearing from users, we've added new views to 1) bulk-publish content when making assessments public, 2) rapidly convert references into studies where data can be extracted or study evaluations can be completed, and 3) create studies for extraction/risk of bias given a PubMed or HERO ID without the intermediate step of making a reference
  • Added custom sort orders for visuals as well as textures to help distinguish data across studies
  • Expansion of the high level assessment metadata, making the objective required and adding a publicly available authors field to describe what users or groups created the assessment

This release also includes a first pass of a revised epidemiology data extraction module. This hasn't been yet publicly released and is only available by a feature flag; we hope to complete next quarter after further evaluation from epidemiologists.

The complete changelog is described below:

New features:

  • Assessment

    • Improved publicly available metadata about the purpose of assessments and authors #574 #600
    • Added page for bulk publishing/unpublishing content #582
    • Paginated assessment list views #611
    • Assessment attachment changes now happen in an inline form #564
    • Removed inactive users from team-members list #601

  • Literature

    • Customizable reference ordering on literature views #571
    • Made order by reference author case insensitive #577
    • Ignore duplicate HERO ids on import #590
    • Ignore duplicate rows when updating reference tags using CSV #594
    • Improved accession number parsing in RIS #609

  • Study

    • Added form to bulk convert references to studies #599
    • Added form and API to create studies directly from a given HERO/PubMed identifier #613

  • Risk of Bias/Study evaluation

    • Removed extraneous ordering arrows for singular domain and metric #605
    • Added metric validation to create API for risk of bias #619
    • Added study filtering to review assignment page #617
    • Added database constraint to make sure risk of bias always has an associated study #593

  • Summary tables and visualizations

    • Added study evaluation tables #581 #607
    • Added custom sorting to data pivots #575
    • Added capability for additional textures #365
    • Added "Other inferences" field to evidence profile tables #627

  • Animal

    • Converted dose tables into a more compact list format #612

  • Epidemiology

  • General user-interface improvements

    • Added popup to extend expiring sessions #623
    • Focus is now given to the first input on forms that use Quill #615
    • Added error notifications on the data cleanup page #576
    • Standardize layout across pages #518 #602
    • New animation on 500 error page #595

  • Administrative

    • Created an admin dashboard framework and new charts/tables #580

  • Client

    • Released a new version of hawc_client #608
      • To update: pip install -U hawc-client==2022.2
    • When attempting to authenticate, client notifies the user if authentication token is invalid #621

Software upgrades/security/stability:

  • Updated python packages #579 #628
  • Refreshed test cassettes #578
  • Replaced Django JSON views with DRF implementation #560
  • Moved admin components into a new hawc_admin app #580
  • Expanded reachable APIs in the OpenAPI schema #588
  • Added a class to hold current and future feature flags #622
  • Added additional logging to APIs #610
  • Updated DSSTox source #572
  • Fixed vulnerability on endpoint copy page #616

Bug/regressions fixed:

  • Fixed generational form display choices #560
  • Fixed dose activation on endpoints #560
  • Fixed layout of untagged references #560
  • Fixed overlap on donut visuals #584
  • Fixed the rendering of BMD values on data pivots #585
  • Fixed rendering of 500 error page #587
  • Fixed errors on API calls of empty tagtrees #591
  • Fixed some errors raised by POST requests with partial data #592
  • Fixed some regressions caused by updated d3 #597
  • Fixed error on visual create page #606
  • Handle deletion of score overrides when study changes warrant deletion #625
  • Fixed pages that show a user's assigned tasks #629
  • Fixed several miscellaneous bugs identified during new site launch #596
  • Removed bulk update task assignment page #624
  • Removed unused API endpoints #589
  • Removed the EXTRA_BRANDING setting #614

Natural Resources - Ecotoxicology - Python
Published by rabstejnek about 3 years ago

Health Assessment Workspace Collaborative - 2021-Q4

January 14, 2022

Big changes this quarter include integrating external authentication systems into HAWC, instead of always requiring to create a custom user account (#525). We also added a much requested bulk-literature tagging application (#519), so you can apply (or remove) tags to dozens of references at once from an Excel spreadsheet, and improved and expanded handling of DOIs throughout the literature application. Finally, we've improved searching and filtering pages for HAWC for greater flexibility (#547), as well more flexible data filtering in creating custom visuals (#520).

The complete changelog is described below:

New features:

  • Authentication

    • Enable integration with external authentication systems Web Access Management (WAM) #525 #541
    • Add optional disclaimer when logging in or creating a new account #517

  • Literature:

    • Added new bulk-tag literature feature from an Excel spreadsheet #514 #519
    • Improved DOI validation and capture from literature searches and imports #533
    • Add DOI to literature exports #526 #530 #558
    • Search and import descriptions now use rich text #537
    • Added "download" actions for all tag views including in visualization popups #549

  • Data extraction

    • Added internal communications fields for assessments #528 #539
    • Added internal communications fields for studies #528 #539
    • Added improved searching/filtering for studies #547
    • Added improved searching/filtering for visuals #547
    • Added improved searching/filtering for assessments #547
    • Enabled copying of data visualizations (not just data pivots) #554
    • Enabled copying of summary tables (not just data pivots) #554

  • Benchmark dose modeling

    • Allow multiple selected models for an endpoint, one per dose-unit #521

  • Data visualization

    • Improve visualization dataset filtering with custom AND, OR, NOT rules #520
    • Improve legend for risk of bias visualizations #550

  • API:

    • Added change audit-logging to API writeable views #524
    • Tie API tokens to user-login sessions #523 #559
    • Add swagger user interface to browse/test the API (admin-only) #545
    • Split study API for the final review and individual risk of bias evaluations #534
    • Add literature tag tree fetch/clone/update methods to API and client #513

  • Development environment

    • Reorganize model choices to move to a constants file #543
    • Improved developer environment with new syncing command #542
    • Code inside the scripts/private path is no longer formatted #540
    • Change default setting for HAWC_LOAD_TEST_DB and add new option 'delete' #536
    • Deployed a new test-environment for ecological data extraction #482

Software upgrades/security/stability

  • Add change audit logs to logging and database for redundancy #546
  • Standardize javascript applications startup in templates #507 #567
  • Convert client module from single file to a python package #562
  • Convert test files into a python package #563
  • Add a flag to conditionally deploy the HAWC admin #522
  • Add sendgrid as a new email provider #548
  • Improved application monitoring by adding container healthchecks #516

Bug/regressions fixed:

  • Fix use of native javascript Set/Map #515
  • Prevent error if unauthenticated user attempts to logout #529
  • Improve data validation for requesting plot downloads #529
  • Fix API bugs #535
  • Fix autocomplete for admin user assignment #538
  • Fix various bugs related to Heatmaps #527
  • Remove eval method in JavaScript #544
  • Minor improvements to test fixture and API #551
  • Fix literature downloads to include child tags #570

Natural Resources - Ecotoxicology - Python
Published by munnsmunns over 3 years ago

Health Assessment Workspace Collaborative - 2021-Q3

October 1, 2021

Our key feature this quarter is a new view of assessment-level change logs for team members. Team members can browse and search the logs to find which objects changed, including the ability to investigate the logs for particular objects. At this point we're not presenting the content which changed, but we may find a way to present it in the future.

In addition, a few additional major updates include:

  • Rewrite the study evaluation/risk of bias user assignment page for speed and flexibility, including the ability to toggle active/inactive status
  • Improve the literature tagging and browsing interface including new search features
  • Documentation on how to build interactive HAWC visuals with an uploaded dataset

The complete changelog is described below:

New features:

  • Overall:
    • Assessment data changelog #486 #497
    • Change permission for making assessments public/editable, based on deployment #484
    • Make accepting user terms of service optional based on deployment #503
  • Literature
    • Enable tag searching with references, including co-occurrence #509
    • Enable changing the DOI on reference form #512
    • Improve tagging user-interface flexibility #510
    • Sort literature references in stable manner #510
  • Study evaluation
    • Update study evaluation user assignment form; enable marking existing reviews active/inactive #495
    • Add new study evaluation score choices consistent with the NTP Report on Carcinogens #511
  • Summary and visualizations
    • Enable/describe HAWC interactivity with uploaded data files #506
    • Improve evidence profile table layout and options #502
    • Improve pagination browsing to select a specific page #508
    • Reorganized visual selector to be more intuitive #498
  • Administrative
    • Improved request logging #487
    • Admin functionality and performance improvements #492 #501 #505

Software upgrades/security/stability:

  • Update python 3.6 to python 3.9; package updates #478 #481
  • Update postgres 9.6 to 12 #478
  • Update javascript build tools and packages #485
  • Refactor build to use standard python wheels instead of a custom approach #477
  • Add javascript unit-testing and integration with CI/CD #501
  • Refactor javascript code to standardize startup/code-splitting, improve editor introspection #477 #491 #504 #505
  • Worker healthcheck for automated checking and plotting #476
  • Revise/update development and documentation #480

Bug/regressions fixed:

  • Add randomness for generating long slugs #502
  • Standardize error messages for assessment uniqueness #490
  • Fix Actions dropdown display on endpoint detail page #500
  • Fix risk of bias score creation signals and defaults #494
  • Make relative URLs absolute to HAWC in a table docx export #483
  • Properly handle SvgRenderer in API #488
  • Heatmap tooltip positioning fix #489
  • Add async task expirations for recurring tasks #493
  • Assorted bugfixes regarding BMD modeling and dose-response confidence intervals #501
  • Fix bugs regarding state where dose-groups do not align with endpoint-groups #505

Natural Resources - Ecotoxicology - Python
Published by shapiromatron over 3 years ago

Health Assessment Workspace Collaborative - 2021-Q2

July 2, 2021

Our key feature this quarter is expanded flexibility and interoperability of the risk of bias / study evaluation module. This module continues to be one of the most frequently used in HAWC, with over 8,500 studies where study evaluation was conducted and over 200,000 individual metric responses. We've added the ability to load an existing approach (instead of a default approach), and added flexibility in which judgment (previously score) options can be used. We've also overhauled the user interface to make it a snappier and more reliable experience.

A few additional notable updates:

  • Expanded flexibility in evidence profile tables and exploratory heatmaps
  • Performance updates (faster study evaluation views, literature pagination, permission checks)
  • Redesign for improved accessibility and consistency with EPA style guides (EPA flavor)

The complete changelog is described below:

New features:

  • Literature screening
    • Add paginated references when showing items on modal #426
    • Better defaults for References with no data available #434
  • Risk of bias/study evaluation
    • Default evaluation approaches are no longer performed and can be selected for each assessment #451
    • Risk of bias domains and study evaluations are now sortable #440
    • Metrics can have customizable judgment response instead of one list for all #441
    • Appropriately show correct study evaluation for override cases on endpoints and outcomes in heatmaps and data pivots #409 #413 #456
    • User interface improvement
      • Rename score to judgment #473
      • Dynamic toggling of metric descriptions #474
    • Make the score/judgment field optional for entry and presentation #463
    • Add timestamps to exports for QA/QC #422
  • Vocabulary
    • Add new APIs for bulk-uploading new terms #417
    • Add new APIs for exporting vocabulary in long and wide forms #439
    • Add diagnostic endpoints for evaluating term usage or issues on assessments #439
    • Require and associate external UIDs for terms #442
  • Exploratory heatmaps
    • Axes are now sortable instead of always alphabetical #449
    • Right-align of text in heatmaps for better layout #458
    • Add observation time to heatmaps #460
    • Better resizing of heatmaps of various sized screens #450
  • Evidence profile table
    • Allow hyperlinks and other rich text in factor-text descriptions #459
    • Add underlines as acceptable table text #420
  • Styling
    • Improve accessibility and 508 compliance alt-text for images #462
    • Add confirmation check when deleting literature tags #428
    • Improved application template styling and flexibility #435 #431 #436 #445 #453 #467 #471
  • Assorted changes
    • Improved performance for permissions testing with assessment checks via permission cache #418
    • Improve help-text on BMD dose-dropping message for #432
    • Update client to show new APIs for creating and modifying epi content #446

Software upgrades/security/stability:

  • Enable HSTS security settings #430
  • Add Referer checking on all safe methods #415
  • Add admin auditing view for browsing media #443
  • Add epi models to admin #448
  • Vendor all js/css instead of external css requests #416
  • Change all URLs from updating to /update/ for consistency #429
  • Refactor and improve code quality for risk of bias application #457 #461 #464 #465 #468 #470 #469 #466
  • Update logging configuration in development #454
  • Make keys-stable in test comparisons to prevent false diffs #472

Bug/regressions fixed:

  • Fix layout recursion on risk of bias heatmaps #438
  • Fix regression where risk of bias table cells when shown on a modal caused a display error #414
  • Fix bug where exploratory heatmap input text prevented adding spaces #444
  • Fix error with updating dosing-regime invalidating endpoint-groups #424
  • Fix validation errors on APIs #421 #427
  • Fix URL validation in evidence profile tables #419
  • Fix test failure requiring remote requests #425
  • Show 404 error for missing dataset revisions #433

Natural Resources - Ecotoxicology - Python
Published by shapiromatron almost 4 years ago

Health Assessment Workspace Collaborative - 2021-Q1

April 1, 2021.

HAWC has long had the ability to build result summary figures in a variety of graphical formats, but tabular options have been limited. This releases introduces HAWC summary tables, which enable users to create summary tables in their assessments. At this point, we've created two different table formats - a generic table, which allows a user to define their own table structure and content, and an evidence profile table, which is a predefined table structure for expert-summarization by evidence type. HAWC tables natively have a web view and an export to Microsoft Word. The current two table formats require new data entry into HAWC; future HAWC table updates will include query-based tables from extracted data in HAWC.

We've also overhauled the literature tagging and browsing view, including a quicksearch feature, making tags hyperlinked, and adding a wordcloud and filterable histogram of results by year. On the backend, we made a few changes to improve performance (aka speed), with a few more in the queue for future updates.

A few additional notable updates:

  • Custom literature tagtrees can be resized for larger visuals with more nodes
  • Exploratory heatmap tables can now be filtered to only include a subset of the data, including risk of bias summaries
  • Epidemiology data in HAWC can now be created/updated via API, allowing data migration from external tools

The full changelog is below:

New features:

  • New table creation features #407
  • Overhaul literature tags, searching, and displays #393
    • Make tag badges interactive and clickable so you can view related References
    • Added a quicksearch to enable searching by title, author, or year
    • Added a filterable histogram by year
    • Added a wordcloud
    • Improved query performance for faster pageviews
  • Visualization updates
    • Literature tagtree height/width are now customizable to prevent tag node edge conflicts #405
      • Evidence heatmap updates #404
      • For custom heatmaps, add the ability to filter the dataset to display only a subset of the data
      • Add new calculated summary fields for bioassay data, consistent with fields in data pivot
      • Add overall study evaluation/risk of bias fields to epi and bioassay data
    • Changed default URLs for visuals to be slugs instead of IDs, consistent with other other pages in HAWC. ID links now redirect to the slug view automatically #401
  • By default, links to public HERO are shown. However a user can modify their user profile to request to see private HERO links. #398
  • Update header footer layout, customizable contact us #389
  • Audit-logging added for delete views #391
  • Add new guardrails for literature reference deletion to prevent unintentional data loss #392
  • Make new editable, cacheable frontmatter and about page #394 #395
  • Added draft ecological data extraction schemas for admin testing #355

New APIs for bulk-modifying content:

  • Epi content can now be created via API #411
  • Simplified reference -> study creation API, making short_citation and full_citation optional, and calculated if not provided, consistent with how studies are created in the form view #396

Software upgrades/security/stability:

  • Improve container layout and deployment documentation #385
  • Add new admin diagnostic action for checking email service #402
  • Added new healthcheck for the bmds2 webservice to periodically check for proper functioning; as well as a diagnostic test in the admin #406
  • Add new optional load of test database for dev/test environments from CI #402
  • Move healthcheck status url outside admin #388
  • Migrated from google analytics to google tag manager; make optional specified by environment variables #399
  • Improve performance (speed) for animal endpoint list, animal groups, and literature tagging #400 #393
  • Support Python 3.9 with pubmed XML parser #412

Bug/regressions fixed:

  • Always show default, non-override scores in data pivot exports #387
  • Fix epi country filter after migration to many to many field #384
  • Restored entry of null-booleans after regression (eg., dosing regime, positive control) #383
  • Fix endpoint list sorting regression #408

List of changes:
https://github.com/shapiromatron/hawc/pulls?q=is%3Apr+is%3Amerged+sort%3Aupdated-desc+closed%3A%3E2021-01-15+created%3A%3C2021-04-01+

Natural Resources - Ecotoxicology - Python
Published by shapiromatron about 4 years ago

Health Assessment Workspace Collaborative - 2020-Q4

January 14, 2021.

A large amount of our time this quarter was spent updating and redesigning the HAWC frontend. The project has grown organically over around eight years, patching new features and functionality as it grew. Unfortunately, we hit a point where the site was looking dated, and there were user-interface inconsistencies, making it difficult to add new features. We've therefore updated from a 2012 framework to a modern one, and added the ability to more easily skin an application with branding.

In addition to the website redesign, a few critical new features were added:

  • References in HAWC can now be manually updated to associate with a Pubmed and/or HERO ID
  • Restore copy buttons for risk of bias/study evaluation text
  • Unaccented author search feature (eg., a search for "fred" returns "Frédéric")
  • Added new programmatic APIs to copy risk of bias across assessments, and associate existing bioassay endpoints with controlled vocabulary terms

Finally, we added the ability to (optionally) restrict assessment creation to a subgroup of users. In some cases organizations may want to limit the content that is added to HAWC, and therefore, if enabled, HAWC admins can assign users to a group that has the ability to create new assessments. Once an assessment is created, members of the assessment can manage just as before.

The full changelog is below:

New features:

  • overhaul look and feel of site, migrating from Bootstrap 2 (2012) to Bootstrap 4 (2018+)
    #334 #335 #336 #337 #338 #338 #343 #345 #346 #347 #348 #354 #356 #358 #359 #360 #361 #362
  • enable users option to specify controlled vocabulary for assessment #326
  • for risk of bias/study evaluation, rename scores -> judgements #331
  • restyle header, footer, and navigation bar #342 #367
  • enable updating any reference to manually add PubmedID or HERO ID after import #364
  • Restore copy buttons for risk of bias/study evaluation text #366
  • (optionally) restrict assessment creation to subgroup of users #369
  • Enable unaccented search for author names #373
  • Improve risk of bias score admin view and identify orphaned override content objects #374

New APIs for bulk-modifying content:

  • add new API to bulk update endpoint term IDs #323
  • allow copying risk of bias/study evaluations across assessments #332
  • add new API for fetching required bioassay metadata #371

Software upgrades/security/stability:

  • upgrade to django 3.1 #324
  • add server-side browser check to possible client incompatibility #339
  • move breadcrumb logic to view instead of template #341
  • re-enable and expand integration tests #350
  • sanitize referer and fix XSS vulnerability #349 #363
  • remove nginx configuration from docker container deploys #352

Bug/regressions fixed:

  • cache EHV vocabulary page: #322
  • restore model ordering defaults #325
  • fix model ordering distinct issues #327
  • fix cog button in visualizations #328
  • update pending deprecation warnings #329
  • remove non-final reviews from rob heatmaps #330
  • fix NA scores on data pivots #333
  • fix tabular export when literature tag names are not unique #368
  • fix url slugs not properly escaping special characters #370

Natural Resources - Ecotoxicology - Python
Published by shapiromatron over 4 years ago

Health Assessment Workspace Collaborative -

October 11, 2020.

This quarter's update has three major new features/updates:

  1. Allowing use of controlled vocabulary for in vivo bioassay endpoint data extraction in HAWC. To take advantage of this new feature, we've also released the Environmental Health Vocabulary (EHV), designed by health scientists at EPA/ORD/CHPEA, this is designed to be a living vocabulary which will be updated and curated frequently as new health assessments extract content in new biological domains (https://hawcproject.org/vocab/ehv/; login required). Controlled vocabularies in HAWC allow mapping of terms to external ontologies, including potentially mapping to adverse outcome pathways (AOP) or other ways of organizing biological causality and critical effect. Use of the vocabulary in HAWC is optional; you can still enter content without using the vocabulary as before.
  2. Integration with the US EPA Chemistry dashboard (https://comptox.epa.gov/dashboard) by using DTXSID as a preferred chemical identifier instead of name/CASRN, which allows for QA/QC of data entry by using this highly curated scientific database. Assessments, bioassay endpoints, invitro endpoints, and epidemiological exposures can now be mapped to DTXSIDs.
  3. Improving software stability by full-stack integration tests in our CI/CD pipeline, which allows for testing visualizations and rich frontend client applications

A full list of changes are available below:

  • Vocabulary

    • Create new application for controlled vocabulary and mapping to related external entities
    • Load v1 of enviro health vocabulary EHV for EPA HAWC assessments, developed by EPA/ORD/CPHEA colleagues
    • Add a display view, /vocab/ehv/ to browse the vocabulary

  • Assessment

    • Add DTXSID integration and modify dashboard to query Chemistry dashboard API
    • Migrate existing CASRN to DTXSID
    • Add assessment-level (and overall) logs
    • Add blog model for reporting HAWC updates
    • Add job-model for asynchronous reports
    • Datasets can now be published or unpublished, just like studies or visuals
    • Rewrite data-extraction cleanup using mobX instead of redux, ignoring controlled vocabulary

  • Literature

    • Added new API endpoint for replacing HERO identifiers
    • Added new API endpoint for updating all reference metadata from HERO
    • Bugfix: update API to check for unique title/slug
    • Update reference page to show all tags currently applied in all views
    • Update tagtree visualization to show/hide nodes with no references
    • Catch common RIS formatting upload errors and report to user as 400s instead of 500s

  • Risk of bias/study evaluation

    • Modify "default" score label to "overall" and only present if a label was not entered
    • Add ability to selectively filter which scores to present in an RoB heatmap

  • Animal bioassay

    • Add DTXSID integration and modify dashboard to query Chemistry Dashboard API
    • Migrate existing CASRN to DTXSID
    • Update bioassay endpoint forms for use of vocabulary, including a quick insert by ID method
    • Add mechanism (admin only) to migrate terms to use the new vocabulary
    • Update API and client to add endpoints using controlled vocabulary

  • Study

    • fixed bug for project managers locking/unlocking studies
    • Show ask_authors text details, preserving whitespace

  • Epi

    • Add DTXSID field to exposure objects in HAWC

  • Management

    • Add ability to unset users in tasks

  • Client

  • Testing

    • Add new integration tests which use a browser to test interactive javascript functionality
    • Bring in external dependencies into hawc (litter_getter and docx_utils)
    • Ensure IE11 error is still displayed on older browsers
    • Fix bugfix on /about/ page and add tests for reliability

  • Development

    • Remove eslint warnings to further standardize javascript frontend application code
    • Update d3 from v3 to v5
    • Generate lines of code report
    • Force ordered querysets in all assessments for deterministic reporting of data

  • Deployment

    • refactor to allow for kubernetes deployment in addition to docker-compose
    • added new /admin/:uuid/healthcheck/ url

Natural Resources - Ecotoxicology - Python
Published by shapiromatron over 4 years ago

Health Assessment Workspace Collaborative -

Released July 15 2020.

This quarter's updates focused on developing new visualizations for displaying data extracted in HAWC or in other tools. Evidence maps created in Tableau can now be embedded into HAWC, or if data has been extracted in HAWC, exploratory heatmaps can be created to view the data. In addition, standardized, commonly used views of data in an assessment are available whenever data has been extracted into HAWC (eg., see the dashboards in PFBS). These new views are available on all new and existing assessments.

Updates were also made in expanding data-capture and display of study evaluation/risk of bias data. In addition, a new assessment-level Dataset data type was added to allow users to upload datasets created in external systems (such as DistillerSR, SWIFT, or Excel) and use in visualizations in HAWC. We anticipate other features will be available with the new Dataset type in future updates.

We've also expanded the API and HAWC client so that studies, study evaluation/risk of bias responses, and animal bioassay data can be created via API in addition to our standard exports of these data. This should allow for easier interoperability with external datasets, databases, and applications.

A full list of changes are available below:

  • Assessment

    • Add new model, Dataset, where tabular datasets can be uploaded and used in HAWC assessments. Datasets are versioned and only the most recent version is available to the public if an assessment is made publicly available. Datasets can currently be used in creating custom exploratory heatmaps, but it is expected that they will be used in the future for other purposes.
    • Added a new assessment option to include data extracted from assessment in an anonymized machine learning (ML) dataset export
    • beta-testing: Added new ML exports using HAWC data

  • Literature

    • Previously, only references tagged with "Included" or child-tag could be extracted in HAWC; this is now a configurable option to allow for more dynamic tag-trees
    • Integration of literature tags with new exploratory heatmap visualizations
    • beta-testing: topic modeling for literature using title and abstract and visualization of topics. Preliminary approach using TFIDF + NMF + TSNE with the scikit-learn library.

  • Study evaluation/Risk of bias

    • Add new (optional) field to all risk of bias/study evaluation - direction of effect
    • Update visualizations to include the direction of effect when entered

  • Visualization

    • Add new custom visualization type - exploratory heatmap. Exploratory heatmaps can be created with data currently extracted in HAWC (literature tags, bioassay extraction, or epi extraction) or with a tabular dataset upload
    • In addition to custom exploratory heatmaps, standard generic heatmap visuals are available for extracted HAWC data. These are preloaded using commonly-used displays of bioassay and epi data (eg., PFBS), and are available for all assessments that have data extracted.
    • Add new visualization type - embedding external visuals. This allows embedding of visuals from external systems; it currently supports the public Tableau website.
    • Data pivot updates:
      • Update user interface (UI) for conditional formatting options in data pivots
      • Fix bug regarding column text alignment with maximum text-size
      • Fix bug, remove dose-groups where data are not extracted for a single Endpoint, even if it was captured in the Dosing Regime
    • Crossview updates:
      • Add endpoint name as a new filter option
    • Study evaluation/risk of bias heatmap
      • Update legend to include footnotes for additional extraction indications
      • Update heatmap to display direction of effect if evaluated
    • beta-testing: new visualization for exploring critical effect levels (LOEL, NOEL, BMD, etc) across all endpoints in an assessment, grouped by system

  • Client

    • Release python client on pypi to allow for easier install (eg., pip install hawc_client)
    • Update python client to create study, risk of bias response, or bioassay data
    • Add a status-bar indicator for large dataset downloads
    • Added a test-suite to Python client for continuous integration testing

  • API

    • Add new endpoints to create risk of bias response
    • Add new endpoints to create study
    • Add new endpoints to create animal bioassay extraction
    • Add new tabular exports, commonly used in exploratory heatmap visuals
    • Add new assessment dataset API views
    • Add OpenAPI API schema definition file (admin-only)

  • Testing

    • Add vcrpy to test suite for faster tests and to remove dependency on external websites like PubMed, HERO, etc.
    • Add new make coverage command to create testing coverage reports
    • Add test-suite for hawc_client and integrate with CI
    • Expanded tests for tabular exports to improve stability, commonly used in visualizations

  • Other

    • Refactor tabular exports to use REST API instead of custom solution
    • Remove printf in javascript in favor of ES6 string literals
    • Large tabular datasets are now cached and responses are reused over a period of hours. If data changes overtime, users now have the option to delete assessment caches and build new.

Natural Resources - Ecotoxicology - Python
Published by shapiromatron almost 5 years ago

Health Assessment Workspace Collaborative - hawc-client 2020.5

First pypi HAWC client library release.

https://pypi.org/project/hawc-client/2020.5/

Natural Resources - Ecotoxicology - Python
Published by shapiromatron almost 5 years ago

Health Assessment Workspace Collaborative - 2020-Q1

March 31st 2020

Multiple updates were added to the literature import and tagging module, as well as more flexibility in the risk of bias/study evaluation section by allowing study-level overrides to specific questions. We've also expanded the visualization libraries being used, making it easier to make new visualizations in future releases. Stability was improved by adding a database fixture for easier testing, updating all packages to recent versions, and adding full windows support to the developer environment.

We also released both a Python and R-based HAWC client. This allows for developers, data scientists, and scientists to interact with both reading and writing data to HAWC via code instead of manually in the user-interface. Expect a dramatic expansion of the API in future releases.

Full changelist below:

  • General

    • Replace chemspider with the EPA chemicals dashboard
    • Add the ability to select language for NOAEL/NOEL/NEL, LOAEL/LOEL/LEL for assessments
    • Updated all packages to latest version for stability and easier development in Windows
    • Update django to version 2.2 LTS

  • Literature

    • Fix imports from HERO using new HERO API
    • Addition validation when importing from HERO to ensure all selected references for import can be imported into HAWC
    • Allow faster querying of PubMed using the new PubMed API key
    • Customizable literature tagtrees - can change which nodes are displayed and filter references by selected tags
    • Preserve UI settings for expanded/collapsed tags when tagging literature
    • Increase maximum PubMed query size to 10,000
    • Update reference detail layout; add related searches/imports for references
    • Show full author list instead of subset; full-author list is now also searchable when finding references
    • Add new histogram for literature overview by year
    • Added new API endpoints for interacting with literature module
      • /lit/api/assessment/{id}/tags/ - view all tags for assessment
      • /lit/api/assessment/{id}/reference-ids/ - view references for assessment and related PubMed/HERO ids
      • /lit/api/assessment/{id}/reference-tags/
        • GET: view references and associated hawc tags for each reference
        • POST: append/replace existing reference-tag relationships with new relationships
      • /lit/api/search/
        • POST: added new create HERO import API for importing references from HERO

  • Risk of bias app

    • Add new risk of bias overrides feature
      • multiple scores can be selected for a single metric/study combination
      • labels can provide context to which data the score are relevant for
      • extracted bioassay or epi components can be associated with overrides
      • updated all visualizations and data views to present override results in addition to default

  • Visualization

    • Add conditional formatting to data pivot reference-bars
      • commonly used to show if a trend-test is significant
    • Wrap-text automatically for long tag-node names in literature trees

  • Client

    • Added new Python-based HAWC client for reading/writing data to HAWC
    • Added new R-based HAWC client for reading/writing data to HAWC

  • API

    • enabled token-based authorization to allow standard automated tooling to access the HAWC API
    • Created new tabular style exports - for tabular REST API, can multiple standard output formats by specifying format=xlsx, or {csv, xlsx, json, html}

  • Testing

    • Add a database test-fixture to dramatically expand unit-testing capabilities
    • Updated test to use fixture instead of creating objects in test-setup
    • Created a new test for asynchronous celery task execution in admin
    • Added tests for phantomjs (svg -> png) conversion stability

  • Developer environment

    • Support Windows as a developer environment for HAWC
    • Updated developer documentation
    • Add new commands to anonymize developer database
    • Simplified developer environment setup to make the settings/local.py file optional
    • Expanded available /admin/ views
    • Update docker deployments to use recent container versions
    • Added new API endpoints for rendering plotly visualizations

Natural Resources - Ecotoxicology - Python
Published by shapiromatron about 5 years ago

Health Assessment Workspace Collaborative - 2019-Q4: hawc merge

We've merged the teamhero/hawc branch into the main branch, and have done a large amount of package updating and refactoring to make the developer environment easier to use and ready to take advantage of recent python improvements including automated formatting and type-checking.

  • Overall changes
    • black + flake8 linting and formatting enforcement
    • Restructed python code into a true python package
    • Update to webpack3 including linting and formatting enforcement
    • Added CI and automated testing
    • Added utility command-line scripts to capture all data pivot visuals SVG and PNG
    • Added server level hawc flavors which allow for different behavior across controls {EPA, PRIME}
      - update admin pages and default views
  • Assessment-level changes
    • Allow nomenclature options at the assessment level (NOEL/NOAEL and LOEL/LOAEL; "Risk of bias" vs "Study evaluation")
      - added more content cleanup options
  • studies
    • added the ability to "lock" studies so they cannot be edited, unless a project-manager unlocks
    • added new options for COI reporting
      - animal bioassay:
      - animal husbandry and diet moved from experiment to animal
      - text displays as NOAEL or NOEL depending on assessment-specific settings
      - animal group endpoint table is now sortable by multiple fields
      - expanded endpoint copy form to allow more richer sorting and filtering
      - epi:
      - Study populations can belong to multiple countries instead of just one
      - Epi exposures can now report multiple "central tendency" descriptions of exposure instead of one
  • risk of bias
    • Allow alternate names for Risk of Bias / Study evaluation
    • Different score-choices available based on hawc "flavor" (epa vs prime)
    • An single overall study confidence metric can be created per reference, if needed
    • Added new selector to toggle if metric is required for invitro data
    • Add assessment-level instructions for conducting evaluation
    • risk of bias/study evaluation scores can be shown/hidden at the assessment level
    • donut plots have domain text rotated
    • metrics can have shorter names to be presented in tables and visuals
    • risk of bias information added to data-pivots for sorting/filtering/displaying

Natural Resources - Ecotoxicology - Python
Published by shapiromatron over 5 years ago

Health Assessment Workspace Collaborative - 2018-Q3

2018 Q3 release.

Natural Resources - Ecotoxicology - Python
Published by shapiromatron over 6 years ago