Data

Tools

Indexes

Applications

Experiments

Open Sustainable Technology

A curated list of open technology projects to sustain a stable climate, energy supply, biodiversity and natural resources.

OrcaHello

A real-time AI-assisted killer whale notification system.
https://github.com/orcasound/aifororcas-livesystem

Category: Biosphere
Sub Category: Marine Life and Fishery

Keywords

artificial-intelligence audio-analysis audio-processing bioacoustics deep-learning inference machine-learning marine marine-biology neural-network orcas real-time realtime whales

Keywords from Contributors

transforms archiving measur observation generic conversion optimize compose language-model animals

Last synced: about 1 hour ago
JSON representation

Repository metadata

Real-time AI-assisted killer whale notification system (model and moderator portal) :star:

README.md

OrcaHello: A real-time AI-assisted killer whale notification system 🎱 🐋

OpenSSF Scorecard

Orcasound maintains a hydrophone network in Puget Sound (near Seattle, WA, USA, Northeast Pacific). Killer whales (aka orcas) often swim by these hydrophones (underwater microphones) and vocalize with a wide range of calls.

Through annual Microsoft hackathons since 2019 and with the volunteer efforts of many heroic Orcasound open source contributors, we have trained and continue to refine a deep learning model to find these calls in live hydrophone audio. The model is at the core of the real time inference system we call OrcaHello which aims to help recover the endangered population of Southern Resident Killer Whale (SRKW) - the iconic orcas that frequently seek salmon in Puget Sound, and also range annually from California to Alaska. OrcaHello is a part of ai4orcas.net, an International effort to apply cutting-edge artificial intelligence to orca conservation.

Learn more about OrcaHello via:

This repository contains the implementations for the following components that make up the OrcaHello real time inference system:

System overview

The diagram below describes the flow of data through OrcaHello and the technologies used.

System Overview

As of September, 2022, the data flow steps include:

  1. Live streaming of audio data via AWS (from Raspberry Pis running orcanode code to Orcaound's S3 open data registry buckets)
  2. Azure-based analysis (via AKS in 2021-2, ICI 2019-2020; ingestion of 10-second segments from S3, inference on 2-second samples using the current OrcaHello binary call classifier, concatenation of raw audio into 60-second WAV files and spectrogram generation)
  3. Moderation of model detections by orca call experts (moderator notification, authentication in moderator portal, annotation and validation)
  4. Notification of confirmed calls from endangered orcas for a wide range of subscribers (researchers, community scientists, managers, outreach/education network nodes, marine mammal observers, dynamic mitigation systems, oil spill response agencies, enforcement agencies, Naval POCs for sonar testing/training situational awareness, etc.)

Each overlapping 2-second data segment is classified as a whale call or / not. Shown below is a 1-minute segment of hydrophone audio visualized as a spectrogram with whale calls detected by the model (delineated by white boundaries).

Detections

When whale activity is detected by the model, it sends an email to our Moderators who are killer whale experts (bioacousticians).

Moderator Email

Once they receive this notification, they can visit the public Moderator Portal shown below to confirm or reject model detections, and to annotate each candidate.

Moderator Portal

Most importantly, they confirm whether or not the whale call was emitted by an endangered Southern Resident Killer Whale (SRKW). If a SRKW is confirmed, notifications are sent to subscribers, like this email message (2022 example):

Subscriber Email

Contributing

You can contribute by

  1. Creating an issue to capture problems with the Moderator Portal and documentation here.
  2. Forking the repo and generating a pull request to fix an issue with the code or documentation.
  3. Joining the Orcasound open source organization on Github to edit the wiki and/or help review pull requests.

To contribute a pull request for a specific subsystem, please read the corresponding contributing guidelines and READMEs.

Current subteams and leads are as follows:

  • Machine Learning and Artificial Intelligence (Lead: Patrick Pastore)
  • Moderation Portal (Lead: Mike Cowan)
  • User Experience (Lead: Claire Goetschel)
  • Notification System (Lead: Skander Mzali)
  • Infrastructure (Lead: Dave Thaler): this subteam deals with Azure and GitHub infrastructure

New volunteers are welcome in all subteams.

General Resources

Project Page - contains information about the system and a brief history of the project.

Related Projects

Owner metadata

GitHub Events

Total
Last Year

Committers metadata

Last synced: 4 days ago

Total Commits: 243
Total Committers: 36
Avg Commits per committer: 6.75
Development Distribution Score (DDS): 0.868

Commits in past year: 40
Committers in past year: 9
Avg Commits per committer in past year: 4.444
Development Distribution Score (DDS) in past year: 0.625

Name Email Commits
Akash Mahajan a****a@m****m 32
micowan m****n@m****m 32
micya m****a@m****m 29
dependabot[bot] 4****] 15
Dave Thaler d****8@g****m 15
catskids3 c****3@g****m 14
Prakruti p****a@g****m 12
Trua Shamu 4****u 12
Prakruti Gogia P****a@m****m 8
Aayush Agrawal a****w@m****m 8
Kenneth Rawlings r****k@g****m 8
aayushmnit a****t@g****m 7
Patrick Pastore p****e@g****m 6
Scott Veirs s****s@g****m 5
Diego Saltijeral j****s@g****m 5
tanviraja24 1****4 4
Adele Bai a****i@m****m 3
Sonu Bhandari s****i@m****m 3
Dmitry Volodin m****e@g****m 2
dyuvaraaj 4****j 2
Bret Nestor b****r@h****m 2
Akash Mahajan a****0@g****m 2
Aarju Kumar a****r 2
Dmitrii Vasilev d****v@m****m 2
Scott Veirs s****t@b****g 2
Ben Leung k****g@m****m 1
Benjamin Chew 4****k 1
Bruno Grande g****1@g****m 1
Daniel Pickens d****s@m****m 1
DanielPickens 7****s 1
and 6 more...

Committer domains:

Issue and Pull Request metadata

Last synced: 1 day ago

Total issues: 117
Total pull requests: 113
Average time to close issues: 7 months
Average time to close pull requests: 28 days
Total issue authors: 23
Total pull request authors: 29
Average comments per issue: 1.73
Average comments per pull request: 1.09
Merged pull request: 92
Bot issues: 0
Bot pull requests: 14

Past year issues: 51
Past year pull requests: 65
Past year average time to close issues: about 1 month
Past year average time to close pull requests: 11 days
Past year issue authors: 11
Past year pull request authors: 10
Past year average comments per issue: 0.98
Past year average comments per pull request: 0.83
Past year merged pull request: 52
Past year bot issues: 0
Past year bot pull requests: 12

More stats: https://issues.ecosyste.ms/repositories/lookup?url=https://github.com/orcasound/aifororcas-livesystem

Top Issue Authors

  • scottveirs (25)
  • clgoetsc (17)
  • micya (16)
  • prakruti (16)
  • dthaler (12)
  • pastorep (5)
  • aserfass-msft (3)
  • bnestor (3)
  • BrunoGrandePhD (2)
  • micowan (2)
  • rawlink (2)
  • veirs (2)
  • dbaing17 (2)
  • ricanddorry (1)
  • anand-raju (1)

Top Pull Request Authors

  • dthaler (22)
  • dependabot[bot] (14)
  • micowan (14)
  • pastorep (9)
  • tanviraja24 (7)
  • micya (6)
  • rawlink (5)
  • BrunoGrandePhD (3)
  • prakruti (3)
  • bnestor (3)
  • dyuvaraaj (3)
  • TruaShamu (2)
  • akashmjn (2)
  • aayushmnit (2)
  • Molkree (2)

Top Issue Labels

  • 2024-hackathon (38)
  • moderator portal (38)
  • enhancement (30)
  • UX-design (20)
  • notification system (19)
  • Azure (17)
  • model training (16)
  • inference system (14)
  • bug (14)
  • automatedretrainingnegsamples (13)
  • 2023-hackathon (10)
  • Addressed by #155 (10)
  • good first issue (6)
  • documentation (5)
  • architecture (3)
  • model evaluation (2)
  • dependencies (2)
  • question (1)
  • maintenance (1)
  • .NET (1)

Top Pull Request Labels

  • dependencies (14)
  • .NET (7)
  • python (6)
  • moderator portal (4)
  • inference system (3)
  • github_actions (2)
  • Azure (1)
  • bug (1)

Dependencies

InferenceSystem/requirements.txt pypi
  • awscli *
  • azure-cosmos *
  • azure-cosmosdb-table *
  • azure-storage-blob *
  • boto3 *
  • bs4 *
  • fastai ==1.0.61
  • ffmpeg-python *
  • ipython *
  • json *
  • librosa ==0.8.0
  • m3u8 *
  • matplotlib *
  • numba ==0.48
  • numpy *
  • opencensus-ext-azure *
  • opencv-python *
  • orca-hls-utils *
  • pandas *
  • pydub ==0.24.1
  • pytz *
  • requests *
  • spacy *
  • torch *
  • torchaudio ==0.6.0
  • torchvision *
  • tqdm *
ModelEvaluation/requirements.txt pypi
  • librosa *
  • matplotlib *
  • numpy *
  • pandas *
  • scikit-learn *
  • seaborn *
  • tabulate *
  • tqdm *
.github/workflows/AIForOrcas.Client.Web.yaml actions
  • actions/cache v2 composite
  • actions/checkout v2 composite
  • actions/setup-dotnet v1 composite
  • azure/webapps-deploy v2 composite
.github/workflows/AIForOrcas.Server.yaml actions
  • actions/cache v2 composite
  • actions/checkout v2 composite
  • actions/setup-dotnet v1 composite
  • azure/webapps-deploy v2 composite
.github/workflows/NotificationSystem.yaml actions
  • actions/cache v2 composite
  • actions/checkout v2 composite
  • actions/setup-dotnet v1 composite
  • azure/functions-action v1 composite
InferenceSystem/Dockerfile docker
  • ubuntu 18.04 build
ModeratorFrontEnd/AIForOrcas/AIForOrcas.Client.Web/wwwroot/vendor/fontawesome-free/package.json npm
ModeratorFrontEnd/AIForOrcas/AIForOrcas.Client.BL/AIForOrcas.Client.BL.csproj nuget
  • Microsoft.AspNetCore.Authorization 3.1.9
  • Microsoft.AspNetCore.Components.Authorization 3.1.9
ModeratorFrontEnd/AIForOrcas/AIForOrcas.Client.Web/AIForOrcas.Client.Web.csproj nuget
  • Blazorade.Msal 2.1.0
  • Blazored.LocalStorage 4.2.0
  • Blazored.Toast 3.2.2
ModeratorFrontEnd/AIForOrcas/AIForOrcas.DTO/AIForOrcas.DTO.csproj nuget
  • Microsoft.AspNetCore.Mvc.Core 2.2.5
  • System.Text.Json 4.7.2
ModeratorFrontEnd/AIForOrcas/AIForOrcas.Server/AIForOrcas.Server.csproj nuget
  • Microsoft.AspNetCore.Authentication.JwtBearer 6.0.8
  • Microsoft.AspNetCore.Authentication.OpenIdConnect 6.0.8
  • Microsoft.EntityFrameworkCore.Cosmos 6.0.8
  • Microsoft.Identity.Web 1.25.2
  • Microsoft.Identity.Web.UI 1.25.2
  • Newtonsoft.Json 13.0.1
  • Swashbuckle.AspNetCore 6.4.0
  • Swashbuckle.AspNetCore.Annotations 6.4.0
ModeratorFrontEnd/AIForOrcas/AIForOrcas.Server.BL/AIForOrcas.Server.BL.csproj nuget
  • Microsoft.EntityFrameworkCore.Cosmos 3.1.9
  • Newtonsoft.Json 13.0.1
  • System.Text.Json 4.7.2
ModeratorFrontEnd/OrcaHello/OrcaHello.Web.Api/OrcaHello.Web.Api.csproj nuget
  • Microsoft.AspNetCore.Authentication.JwtBearer 7.0.10
  • Microsoft.AspNetCore.Authentication.OpenIdConnect 7.0.10
  • Microsoft.AspNetCore.OpenApi 7.0.10
  • Microsoft.Azure.Cosmos 3.35.3
  • Microsoft.Identity.Web 2.13.3
  • Microsoft.Identity.Web.UI 2.13.3
  • Swashbuckle.AspNetCore 6.5.0
  • Swashbuckle.AspNetCore.Annotations 6.5.0
  • System.Linq 4.3.0
ModeratorFrontEnd/OrcaHello/OrcaHello.Web.Api.Tests.Acceptance/OrcaHello.Web.Api.Tests.Acceptance.csproj nuget
  • MSTest.TestAdapter 2.2.10
  • MSTest.TestFramework 2.2.10
  • Microsoft.NET.Test.Sdk 17.6.0
  • coverlet.collector 3.2.0
ModeratorFrontEnd/OrcaHello/OrcaHello.Web.Api.Tests.Unit/OrcaHello.Web.Api.Tests.Unit.csproj nuget
  • MSTest.TestAdapter 2.2.10
  • MSTest.TestFramework 2.2.10
  • Microsoft.NET.Test.Sdk 17.6.0
  • Moq 4.20.69
  • coverlet.collector 3.2.0
ModeratorFrontEnd/OrcaHello/OrcaHello.Web.Shared/OrcaHello.Web.Shared.csproj nuget
  • Microsoft.Azure.Cosmos 3.35.3
  • Microsoft.Extensions.Logging 7.0.0
  • Swashbuckle.AspNetCore.Annotations 6.5.0
NotificationSystem/NotificationSystem.csproj nuget
  • Azure.Storage.Queues 12.11.1
  • Microsoft.Azure.WebJobs 3.0.33
  • Microsoft.Azure.WebJobs.Extensions 4.0.1
  • Microsoft.Azure.WebJobs.Extensions.CosmosDB 3.0.10
  • Microsoft.Azure.WebJobs.Extensions.Http 3.2.0
  • Microsoft.Azure.WebJobs.Extensions.SendGrid 3.0.2
  • Microsoft.Azure.WebJobs.Extensions.Storage 4.0.5
  • Microsoft.NET.Sdk.Functions 4.1.3
  • System.Net.NameResolution 4.3.0
ModelTraining/requirements.txt pypi
  • fastai ==1.0.61
  • ipython *
  • librosa ==0.8.0
  • numpy *
  • pandas *
  • pydub ==0.24.1
  • spacy *
  • torchaudio ==0.6.0
.github/workflows/OrcaHello.Web.Api.yaml actions
  • actions/cache 8492260343ad570701412c2f464a5877dc76bace composite
  • actions/checkout ee0669bd1cc54295c223e0bb666b733df41de1c5 composite
  • actions/setup-dotnet 608ee757cfcce72c2e91e99aca128e0cae67de87 composite
  • azure/webapps-deploy 4bfb30bef2c330e36be280cb1e5726d0fac06233 composite
  • step-security/harden-runner 63c24ba6bd7ba022e95695ff85de572c04a18142 composite
.github/workflows/OrcaHello.Web.UI.yaml actions
  • actions/cache 8492260343ad570701412c2f464a5877dc76bace composite
  • actions/checkout ee0669bd1cc54295c223e0bb666b733df41de1c5 composite
  • actions/setup-dotnet 608ee757cfcce72c2e91e99aca128e0cae67de87 composite
  • azure/webapps-deploy 4bfb30bef2c330e36be280cb1e5726d0fac06233 composite
  • step-security/harden-runner 63c24ba6bd7ba022e95695ff85de572c04a18142 composite
.github/workflows/scorecard.yml actions
  • actions/checkout b4ffde65f46336ab88eb53be808477a3936bae11 composite
  • actions/upload-artifact 97a0fba1372883ab732affbe8f94b823f91727db composite
  • github/codeql-action/upload-sarif 1b1aada464948af03b950897e5eb522f92603cc2 composite
  • ossf/scorecard-action 0864cf19026789058feabb7e87baa5f140aac736 composite
  • step-security/harden-runner 63c24ba6bd7ba022e95695ff85de572c04a18142 composite
ModeratorFrontEnd/OrcaHello/OrcaHello.Console.DataMigration/OrcaHello.Console.DataMigration.csproj nuget
  • Microsoft.Azure.Cosmos 3.35.4
  • Microsoft.Extensions.Configuration 7.0.0
  • Microsoft.Extensions.Configuration.UserSecrets 7.0.0
  • Microsoft.Extensions.DependencyInjection 7.0.0
ModeratorFrontEnd/OrcaHello/OrcaHello.Web.UI/OrcaHello.Web.UI.csproj nuget
  • Blazorade.Msal 2.2.0
  • Radzen.Blazor 4.20.0
  • System.IdentityModel.Tokens.Jwt 7.1.2
ModeratorFrontEnd/OrcaHello/OrcaHello.Web.UI.Tests.Unit/OrcaHello.Web.UI.Tests.Unit.csproj nuget
  • MSTest.TestAdapter 2.2.10
  • MSTest.TestFramework 2.2.10
  • Microsoft.NET.Test.Sdk 17.7.1
  • Moq 4.20.69
  • coverlet.collector 3.2.0
ModelTraining/requirements.lock.txt pypi
  • Bottleneck ==1.4.0
  • Jinja2 ==3.1.4
  • MarkupSafe ==2.1.5
  • PyYAML ==6.0.2
  • Pygments ==2.18.0
  • annotated-types ==0.7.0
  • asttokens ==2.4.1
  • attrs ==24.2.0
  • audioread ==3.0.1
  • backcall ==0.2.0
  • beautifulsoup4 ==4.12.3
  • bleach ==6.1.0
  • blis ==0.7.11
  • catalogue ==2.0.10
  • certifi ==2024.8.30
  • cffi ==1.17.1
  • charset-normalizer ==3.4.0
  • click ==8.1.7
  • cloudpathlib ==0.20.0
  • comm ==0.2.2
  • confection ==0.1.5
  • contourpy ==1.1.1
  • cycler ==0.12.1
  • cymem ==2.0.8
  • debugpy ==1.8.8
  • decorator ==5.1.1
  • defusedxml ==0.7.1
  • docopt ==0.6.2
  • executing ==2.1.0
  • fastai ==1.0.61
  • fastjsonschema ==2.20.0
  • fastprogress ==1.0.3
  • fire ==0.7.0
  • fonttools ==4.54.1
  • future ==1.0.0
  • idna ==3.10
  • importlib_metadata ==8.5.0
  • importlib_resources ==6.4.5
  • ipykernel ==6.29.5
  • ipython ==8.12.3
  • jedi ==0.19.1
  • joblib ==1.4.2
  • jsonschema ==4.23.0
  • jsonschema-specifications ==2023.12.1
  • jupyter_client ==8.6.3
  • jupyter_core ==5.7.2
  • jupyterlab_pygments ==0.3.0
  • kiwisolver ==1.4.7
  • langcodes ==3.4.1
  • language_data ==1.2.0
  • lazy_loader ==0.4
  • librosa ==0.10.0
  • llvmlite ==0.41.1
  • marisa-trie ==1.2.1
  • markdown-it-py ==3.0.0
  • matplotlib ==3.7.5
  • matplotlib-inline ==0.1.7
  • mdurl ==0.1.2
  • mistune ==3.0.2
  • msgpack ==1.1.0
  • murmurhash ==1.0.10
  • nbclient ==0.10.0
  • nbconvert ==7.16.4
  • nbformat ==5.10.4
  • nest-asyncio ==1.6.0
  • numba ==0.58.1
  • numexpr ==2.8.6
  • numpy ==1.24.4
  • nvidia-ml-py3 ==7.352.0
  • packaging ==24.2
  • pandas ==2.0.3
  • pandocfilters ==1.5.1
  • parso ==0.8.4
  • pexpect ==4.9.0
  • pickleshare ==0.7.5
  • pillow ==10.4.0
  • pipreqs ==0.5.0
  • pkgutil_resolve_name ==1.3.10
  • platformdirs ==4.3.6
  • pooch ==1.8.2
  • preshed ==3.0.9
  • prompt_toolkit ==3.0.48
  • psutil ==6.1.0
  • ptyprocess ==0.7.0
  • pure_eval ==0.2.3
  • pycparser ==2.22
  • pydantic ==2.9.2
  • pydantic_core ==2.23.4
  • pydub ==0.24.1
  • pyparsing ==3.1.4
  • python-dateutil ==2.9.0.post0
  • pytz ==2024.2
  • pyzmq ==26.2.0
  • referencing ==0.35.1
  • requests ==2.32.3
  • rich ==13.9.4
  • rpds-py ==0.20.1
  • scikit-learn ==1.3.2
  • scipy ==1.10.1
  • shellingham ==1.5.4
  • six ==1.16.0
  • smart-open ==7.0.5
  • soundfile ==0.12.1
  • soupsieve ==2.6
  • soxr ==0.3.7
  • spacy ==3.7.5
  • spacy-legacy ==3.0.12
  • spacy-loggers ==1.0.5
  • srsly ==2.4.8
  • stack-data ==0.6.3
  • termcolor ==2.4.0
  • thinc ==8.2.5
  • threadpoolctl ==3.5.0
  • tinycss2 ==1.4.0
  • torch ==1.6.0
  • torchaudio ==0.6.0
  • torchvision ==0.7.0
  • tornado ==6.4.1
  • tqdm ==4.67.0
  • traitlets ==5.14.3
  • typer ==0.13.0
  • typing_extensions ==4.12.2
  • tzdata ==2024.2
  • urllib3 ==2.2.3
  • wasabi ==1.1.3
  • wcwidth ==0.2.13
  • weasel ==0.4.1
  • webencodings ==0.5.1
  • wrapt ==1.16.0
  • yarg ==0.1.9
  • zipp ==3.20.2

Score: 8.345692873253867