ODM - Version 3.0.0

Version 3.0.0 of the PHES-ODM represents a major structural expansion compared to version 2.2.3, increasing the model to 19 report tables and 7 look-up tables. The primary differences involve the introduction of new tables for public health actions and data processing, significant refinements to existing attributes, and a massive expansion of genomic and environmental descriptors.

1. New Table Structures

Version 3.0.0 introduces several dedicated tables that were not present in version 2.2.3:

• Public Health Actions (phActions): This table was created to record specific interventions and designations, such as travel bans, mask mandates, and stay-at-home orders. Consequently, the "outbreak" marker was moved from a measure attribute to this dedicated table, and the original outbreak class was depreciated.
• Calculations (calculations): This new table tracks the nature of data treatments, recording the specific equations and standards (e.g., normalization to PMMoV or wastewater flow) applied to raw data.
• Accessions (accessions): Introduced to track external data IDs, specifically for linking genetic sequences to large external repositories like the European Nucleotide Archive (ENA) or GenBank.
• Polygon Relationships (polygonRelationships): This table allows users to define logical connections between geographic areas, such as whether two polygons are neighboring or overlapping.

2. Attribute Refinements and Relational Changes

Version 3.0.0 modified several existing parts to improve clarity and reduce technical conflicts in databases:

• Part ID Changes: The attribute derived was renamed to derivedSamp to better specify its relationship to samples. Similarly, the time part type was renamed to timeClass to avoid collisions with standard "time" data type names.
• Foreign Key Removal: To streamline the model, datasetID was removed as a foreign key from the organizations and instruments tables.

3. Expanded Dictionary Content

Version 3.0.0 significantly increased the number of "building block" parts in the dictionary:

• Bioinformatics Library: v3.0.0 added dozens of parts for specific sequencing and quality control software, such as DADA2, Minimap2, Kraken2, and AdapterRemoval, allowing for more detailed provenance of genomic data.
• Site Aggregation Metadata: New attributes like siteLevel were added to define the scale of a sampling location, allowing users to categorize sites by whether they represent a single building, a neighborhood, or an entire municipality.
• New Environmental Measures: Support was added for new physical and chemical properties, including dissolved oxygen (dissOxy) and oxidation-reduction potential (orp), alongside new units like the practical salinity unit (psu).
• Population Data Intervals: To handle sensitive or approximate data, version 3.0.0 introduced categorical ranges for population counts, such as <10 people or 100,000 - 1,000,000 people.

For additional information on additions and how to use them, please read up on the documentation website, and stay tuned for additional written and video content.

Natural Resources - Water Supply and Quality - Python
Published by mathew-thomson 5 months ago

ODM - Version 2.2.3

A patch release to address lingering bugs and issues related to the updates in the version 2.2.1 and 2.2.2 releases. This release also adds:

1. Support for the new "Supersites Template" - an updated version of the airport template used by the EU, this template is now updated and designed for use at EU "supersites" or major transit hubs (ie. harbours, ports, airports, train and bus stations, etc.). This template focuses largely on the four major priority pathogens: SARS-CoV-2, Influenza, Mpox, and Dengue.
2. Two additional data license categories (ccbyncsa40 - Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, and ccbync40 - Creative Commons NonCommercial 4.0 International License) at the request of users.
3. Additional values to better align and facilitate mapping between the ODM and PHA4GE data formats (more on this in the subsequent version).

This version also corresponds to the new SQL structure, which will hopefully be shared soon.

Natural Resources - Water Supply and Quality - Python
Published by mathew-thomson over 1 year ago

ODM - Version 2.2.2

A patch release to address issue #269 and #270 - namely:

1. missingnessSets in the ERD is missingnessSet in the CSV – these should be aligned one way or another. For this issue, the ERD was incorrect. The parts list header is indeed missingnessSet. However, that should be the header, and missingnessSets should be the part type. This follows the same structure for other headers within the parts list that can also be part types, like aggregationSet (header) vs aggregationSets (part type), etc.
2. There are additional duplicate keys in one of the csv versions of the parts table (ODM_parts.csv, not the ODM_parts_2.2.1.csv):
   f456l
   v1104l
   jn17V
   jn1V
   d339h
   l455s
   flirtSLipGrp

And there are new duplicate keys in the wideNames table:
si_siteID
si_sampleShed
si_popServ
sm_sampleID
sm_collType
sm_pooled
mr_aDateStart
mr_aDateEnd
mr_fraction
sm_saMaterial
pt_refLink
mr_measure
wat_sa_hFr_hMe_hUn_hAg_NR_value
mr_unit
mr_aggregation
wat_si_NA_cod_mgL_me_NR_value
wat_si_NA_bod5t_mgL_me_NR_value
wat_si_NA_cond_uSCm_me_NR_value
wat_si_NA_ph_unitless_me_NR_value
wat_si_NA_tn_mgL_me_NR_value
wat_sa_NA_lodSewa_gc_sin_NR_value
wat_sa_NA_loq_gc_sin_NR_value
mr_reportDate
co_email
mr_notes

These came about from the new HPAIV H5N1 template, which has many overlapping fields with the airports template. Technically we should keep single keys from airplanes, so I've done that here.

Natural Resources - Water Supply and Quality - Python
Published by mathew-thomson over 1 year ago

ODM - Version 2.2.1

This is a patch release to correct an issue in the Zones and Parts tables of the version 2.2.0 release. See issue #267 for additional details.

Natural Resources - Water Supply and Quality - Python
Published by mathew-thomson almost 2 years ago

ODM - Version 2.2.0

This is the official release for v2.2.0 of the Public Health and Environmental Surveillance Open Data Model (PHES-ODM). It includes additional provisions for AMR surveillance, as well as improved reporting for a larger diversity of pathogens, and organism reporting. We added additional mutations and variants (including the current variants of concern - FLiRT and SLip) of the SARS-CoV-2 virus, as well for emerging bird flu (H5N1) markers. Also added additional provisions to improving mapping and interoperability between the USCDC-NWSS data format and the ODM. Additional reporting metrics for protocols and standard curves were also added. Some clean up of the languages table was also done to reduce duplication and remove SQL errors. There are new parts to debug certain issues and respond to demands among the user-base. The largest change is the addition of a new look-up table: wideNames. This helps to standardize and record the wide name formatting to ensure we remain consistent in our wide-to-long and long-to-wide transformations. Additional protocol information was added, and duplication in fields in the languages table were removed.

Natural Resources - Water Supply and Quality - Python
Published by mathew-thomson about 2 years ago

ODM - Version 2.1.0

This is the official release for v2.1.0 of the Public Health and Environmental Surveillance Open Data Model (PHES-ODM). It includes the addition a few new parts as a way to debug issues, or respond to new demands among the user-base. The largest change is the addition of two new look-up tables: countries and zones. These help to standardize location information to the ISO-2 and ISO-3 and to reduce ambiguity and free-text errors when populating these fields, all while improving interoperability of location metadata. Documentation has also been updated to add more information on protocols. Protocols will be the main focus of the next release.

Natural Resources - Water Supply and Quality - Python
Published by mathew-thomson about 3 years ago

ODM - Version 2.0.0

This is the official release for v2.0.0 of the Public Health and Environmental Surveillance Open Data Model (PHES-ODM). It includes the addition of provisions for more detailed reporting of methods, measures for mutations and variants, measures for additional pathogens besides SARS-CoV-2, more detailed metadata, and improved reporting of quality measures. As this version includes a number of massive changes, additional updates with updated version numbers are likely to follow hot on the heels of this release.

Natural Resources - Water Supply and Quality - Python
Published by mathew-thomson about 3 years ago

ODM - Initial stable release

Version 1.0.0!. The ODM development team commits to maintaining a stable version of the ODM. Future breaking changes will be made with care and after public discussion.

See changelog.md for addtions and bug fixes from previous versions.

Natural Resources - Water Supply and Quality - Python
Published by DougManuel almost 5 years ago