{"id":20235,"name":"deep-smoke-machine","description":"Deep learning models and dataset for recognizing industrial smoke emissions.","url":"https://github.com/CMU-CREATE-Lab/deep-smoke-machine","last_synced_at":"2026-04-13T05:03:24.794Z","repository":{"id":41299080,"uuid":"184317473","full_name":"CMU-CREATE-Lab/deep-smoke-machine","owner":"CMU-CREATE-Lab","description":"Deep learning models and dataset for recognizing industrial smoke emissions","archived":false,"fork":false,"pushed_at":"2024-05-01T18:38:47.000Z","size":296119,"stargazers_count":126,"open_issues_count":2,"forks_count":28,"subscribers_count":11,"default_branch":"master","last_synced_at":"2026-03-13T23:36:58.356Z","etag":null,"topics":["air-quality","citizen-science","computer-vision","deep-learning","machine-learning","neural-network","python","pytorch","smoke","smoke-recognition"],"latest_commit_sha":null,"homepage":"http://smoke.createlab.org","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CMU-CREATE-Lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-04-30T19:10:41.000Z","updated_at":"2026-03-09T07:20:30.000Z","dependencies_parsed_at":"2024-05-02T05:09:47.628Z","dependency_job_id":null,"html_url":"https://github.com/CMU-CREATE-Lab/deep-smoke-machine","commit_stats":{"total_commits":526,"total_committers":2,"mean_commits":263.0,"dds":"0.12357414448669202","last_synced_commit":"048d3fd4d914bfcf6b1303cfa7168e2a25fdf4f2"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/CMU-CREATE-Lab/deep-smoke-machine","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CMU-CREATE-Lab%2Fdeep-smoke-machine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CMU-CREATE-Lab%2Fdeep-smoke-machine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CMU-CREATE-Lab%2Fdeep-smoke-machine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CMU-CREATE-Lab%2Fdeep-smoke-machine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CMU-CREATE-Lab","download_url":"https://codeload.github.com/CMU-CREATE-Lab/deep-smoke-machine/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CMU-CREATE-Lab%2Fdeep-smoke-machine/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30677272,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-18T19:30:02.133Z","status":"ssl_error","status_checked_at":"2026-03-18T19:25:53.511Z","response_time":104,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"owner":{"login":"CMU-CREATE-Lab","name":"CMU CREATE Lab","uuid":"3278646","kind":"organization","description":null,"email":null,"website":"cmucreatelab.org","location":"Pittsburgh, PA, USA","twitter":null,"company":null,"icon_url":"https://avatars.githubusercontent.com/u/3278646?v=4","repositories_count":170,"last_synced_at":"2024-04-14T01:17:50.319Z","metadata":{"has_sponsors_listing":false},"html_url":"https://github.com/CMU-CREATE-Lab","funding_links":[],"total_stars":338,"followers":10,"following":0,"created_at":"2022-11-04T09:04:34.236Z","updated_at":"2024-04-14T01:18:23.500Z","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CMU-CREATE-Lab","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CMU-CREATE-Lab/repositories"},"packages":[],"commits":{"id":1254214,"full_name":"CMU-CREATE-Lab/deep-smoke-machine","default_branch":"master","total_commits":526,"total_committers":2,"total_bot_commits":0,"total_bot_committers":0,"mean_commits":263.0,"dds":0.12357414448669202,"past_year_total_commits":0,"past_year_total_committers":0,"past_year_total_bot_commits":0,"past_year_total_bot_committers":0,"past_year_mean_commits":0.0,"past_year_dds":0.0,"last_synced_at":"2026-03-03T22:22:23.326Z","last_synced_commit":"048d3fd4d914bfcf6b1303cfa7168e2a25fdf4f2","created_at":"2023-03-27T10:59:47.821Z","updated_at":"2026-03-03T22:22:23.170Z","committers":[{"name":"Yen-Chia Hsu","email":"hsu.yenchia@gmail.com","login":"yenchiah","count":461},{"name":"SeanPrendi","email":"sean.prendi@gmail.com","login":"SeanPrendi","count":65}],"past_year_committers":[],"commits_url":"https://commits.ecosyste.ms/api/v1/hosts/GitHub/repositories/CMU-CREATE-Lab%2Fdeep-smoke-machine/commits","host":{"name":"GitHub","url":"https://github.com","kind":"github","last_synced_at":"2026-03-05T00:00:32.724Z","repositories_count":6185109,"commits_count":930750866,"contributors_count":36046909,"owners_count":1146610,"icon_url":"https://github.com/github.png","host_url":"https://commits.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://commits.ecosyste.ms/api/v1/hosts/GitHub/repositories"}},"issues_stats":{"full_name":"CMU-CREATE-Lab/deep-smoke-machine","html_url":"https://github.com/CMU-CREATE-Lab/deep-smoke-machine","last_synced_at":"2026-03-03T09:02:50.769Z","status":"error","issues_count":12,"pull_requests_count":3,"avg_time_to_close_issue":6999226.5,"avg_time_to_close_pull_request":18.333333333333332,"issues_closed_count":10,"pull_requests_closed_count":3,"pull_request_authors_count":1,"issue_authors_count":11,"avg_comments_per_issue":2.6666666666666665,"avg_comments_per_pull_request":0.0,"merged_pull_requests_count":3,"bot_issues_count":0,"bot_pull_requests_count":0,"past_year_issues_count":1,"past_year_pull_requests_count":0,"past_year_avg_time_to_close_issue":null,"past_year_avg_time_to_close_pull_request":null,"past_year_issues_closed_count":0,"past_year_pull_requests_closed_count":0,"past_year_pull_request_authors_count":0,"past_year_issue_authors_count":1,"past_year_avg_comments_per_issue":0.0,"past_year_avg_comments_per_pull_request":null,"past_year_bot_issues_count":0,"past_year_bot_pull_requests_count":0,"past_year_merged_pull_requests_count":0,"created_at":"2023-05-09T10:37:01.824Z","updated_at":"2026-03-03T09:02:50.769Z","repository_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/repositories/CMU-CREATE-Lab%2Fdeep-smoke-machine","issues_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/repositories/CMU-CREATE-Lab%2Fdeep-smoke-machine/issues","issue_labels_count":{},"pull_request_labels_count":{},"issue_author_associations_count":{"NONE":12},"pull_request_author_associations_count":{"CONTRIBUTOR":3},"issue_authors":{"dariogonle":2,"Gmlearner231":1,"etatbak":1,"changyangmimeng":1,"fusang1337":1,"Xujxyang":1,"sharontaozi":1,"Xie-Yehui":1,"greenman-ta":1,"Goolo2":1,"10652835":1},"pull_request_authors":{"yenchiah":3},"host":{"name":"GitHub","url":"https://github.com","kind":"github","last_synced_at":"2026-03-05T00:00:11.008Z","repositories_count":13585366,"issues_count":34980686,"pull_requests_count":114008152,"authors_count":11182844,"icon_url":"https://github.com/github.png","host_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/repositories","owners_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/owners","authors_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/authors"},"past_year_issue_labels_count":{},"past_year_pull_request_labels_count":{},"past_year_issue_author_associations_count":{"NONE":1},"past_year_pull_request_author_associations_count":{},"past_year_issue_authors":{"greenman-ta":1},"past_year_pull_request_authors":{},"maintainers":[],"active_maintainers":[]},"events":{"total":{"IssuesEvent":1,"WatchEvent":11},"last_year":{"WatchEvent":4}},"keywords":["air-quality","citizen-science","computer-vision","deep-learning","machine-learning","neural-network","python","pytorch","smoke","smoke-recognition"],"dependencies":[],"score":5.545177444479562,"created_at":"2023-09-11T14:52:11.044Z","updated_at":"2026-04-13T05:03:24.796Z","avatar_url":"https://github.com/CMU-CREATE-Lab.png","language":"Python","category":"Emissions","sub_category":"Emission Observation and Modeling","monthly_downloads":0,"total_dependent_repos":0,"total_dependent_packages":0,"readme":"# deep-smoke-machine\nDeep learning models and dataset for recognizing industrial smoke emissions. The videos are from the [smoke labeling tool](https://github.com/CMU-CREATE-Lab/video-labeling-tool). The code in this repository assumes that Ubuntu 18.04 server is installed. The code is released under the BSD 3-clause license, and the dataset is released under the Creative Commons Zero (CC0) license. If you found this dataset and the code useful, we would greatly appreciate it if you could cite our paper below:\n\nYen-Chia Hsu, Ting-Hao (Kenneth) Huang, Ting-Yao Hu, Paul Dille, Sean Prendi, Ryan Hoffman, Anastasia Tsuhlares, Jessica Pachuta, Randy Sargent, and Illah Nourbakhsh. 2021. Project RISE: Recognizing Industrial Smoke Emissions. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 2021). https://arxiv.org/abs/2005.06111\n\n\u003e [!WARNING]\n\u003e There was an error in implementing the non-local blocks when we wrote the paper. We are very sorry about that. The problem has been fixed in the code in this repository. However, the result of model RGB-NL in Table 7 in the paper is incorrect. We ran the experiment for the RGB-NL model again and submit an [updated version of the paper to arXiv](https://arxiv.org/abs/2005.06111).\n\n![This figure shows different types of videos (high-opacity smoke, low-opacity smoke, steam, and steam with smoke).](back-end/data/dataset/2020-02-24/smoke-type.gif)\n\nThe following figures show how the [I3D model](https://arxiv.org/abs/1705.07750) recognizes industrial smoke. The heatmaps (red and yellow areas on top of the images) indicate where the model thinks have smoke emissions. The examples are from the testing set with different camera views, which means that the model never sees these views at the training stage. These visualizations are generated by using the [Grad-CAM](https://arxiv.org/abs/1610.02391) technique. The x-axis indicates time.\n\n![Example of the smoke recognition result.](back-end/data/dataset/2020-02-24/0-1-2019-01-17-6007-928-6509-1430-180-180-3906-1547732890-1547733065-grad-cam.png)\n\n![Example of the smoke recognition result.](back-end/data/dataset/2020-02-24/0-7-2019-01-11-3544-899-4026-1381-180-180-7891-1547236155-1547236330-grad-cam.png)\n\n![Example of the smoke recognition result.](back-end/data/dataset/2020-02-24/1-0-2018-08-24-3018-478-3536-996-180-180-8732-1535140050-1535140315-grad-cam.png)\n\n### Table of Content\n- [Install NVIDIA drivers, CUDA, and cuDNN](#install-nvidia)\n- [Setup this tool](#setup-tool)\n- [Prepare data](#prepare-data)\n- [Train and test models](#train-test-models)\n- [Code infrastructure](#code-infrastructure)\n- [Dataset](#dataset)\n- [Pretrained models](#pretrained-models)\n- [Deploy models to recognize smoke](#deploy-models-to-recognize-smoke)\n- [Administrator only tasks](#admin)\n- [Acknowledgements](#acknowledgements)\n\n# \u003ca name=\"install-nvidia\"\u003e\u003c/a\u003eInstall NVIDIA drivers, CUDA, and cuDNN\n\nThis installation guide assusmes that you are using the Ubuntu 22.04 server version (not desktop version) operating system.\n\n## Disable the nouveau driver\n\nRun the following on open the file (assuming that you use `vim` as the text editor):\n```sh\nsudo vim /etc/modprobe.d/blacklist.conf\n```\n\nThen, add the following to the file to blacklist nouveau driver:\n```sh\n# Blacklist nouveau driver (for NVIDIA driver installation)\nblacklist nouveau\nblacklist lbm-nouveau\noptions nouveau modeset=0\nalias nouveau off\nalias lbm-nouveau off\n```\n\nNext, regenerate the initial ram file system of the kernel and reboot the computer:\n```sh\nsudo update-initramfs -u\nsudo reboot now\n```\n\nThen, check if nouveau is disabled correctly using the following commands. You should not see any outputs from the terminal when running these commands.\n```sh\nlsmod | grep -i nouveau\ndpkg -l | grep -i nouveau\n```\n\n## Install CUDA and the NVIDIA driver\nWe need to remove old NVIDIA drivers before installing the new one. For drivers that are installed using the files that are downloaded from NVIDIA website, run the following:\n```sh\n# For drivers that are installed from NVIDIA website file, remove the driver using the following command:\nsudo nvidia-uninstall\n```\n\nFor drivers that are installed using `sudo apt-get`, run the folloing:\n```sh\n# For drivers that are installed using sudo apt-get, , remove the driver using the following commands:\nsudo apt-get remove --purge '^nvidia-.*'\nsudo apt-get autoremove\n```\n\nWe also need to make sure that the following packages are installed:\n```sh\nsudo apt-get install build-essential\nsudo apt-get install linux-headers-$(uname -r)\n```\n\nThen, we can install the NVIDIA drivers while installing [CUDA](https://developer.nvidia.com/cuda-toolkit). Different versions of CUDA can be found [here](https://developer.nvidia.com/cuda-toolkit-archive). We use CUDA 11.8 for this project.\n```sh\nwget https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda_11.8.0_520.61.05_linux.run\nsudo sh cuda_11.8.0_520.61.05_linux.run\n```\n\nFollow the instruction on the terminal to install CUDA. Once it is done, run the following commands to add the CUDA runtime library.\n```sh\nsudo bash -c \"echo /usr/local/cuda/lib64/ \u003e /etc/ld.so.conf.d/cuda.conf\"\nsudo ldconfig\n```\n\nThen, open the `.bashrc` file and add the CUDA paths:\n```sh\nvim ~/.bashrc\n\n# Add the following lines to the file\nexport PATH=/usr/local/cuda/bin:$PATH\nexport LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH\n```\n\nNext, reboot the machine:\n```sh\nsudo reboot now\n```\n\nCheck if NVIDIA driver is installed. You should see something on the terminal when running the following commands.\n```sh\nsudo nvidia-smi\nlsmod | grep -i nvidia\nlspci | grep -i nvidia\n```\n\nOptinoally, you can also check CUDA installation by running the following command.\n```sh\nnvcc -V\n```\n\nFinally, install cuDNN. Documentation can be found [here](https://docs.nvidia.com/deeplearning/cudnn/latest/installation/linux.html). Visit [the NVIDIA cuDNN page](https://developer.nvidia.com/cudnn) to download and install cuDNN. Below are the commands from the cuDNN downloading link:\n```sh\nwget https://developer.download.nvidia.com/compute/cudnn/9.1.0/local_installers/cudnn-local-repo-ubuntu2204-9.1.0_1.0-1_amd64.deb\nsudo dpkg -i cudnn-local-repo-ubuntu2204-9.1.0_1.0-1_amd64.deb\nsudo cp /var/cudnn-local-repo-ubuntu2204-9.1.0/cudnn-*-keyring.gpg /usr/share/keyrings/\nsudo apt-get update\nsudo apt-get -y install cudnn-cuda-11\n```\n\n# \u003ca name=\"setup-tool\"\u003e\u003c/a\u003eSetup this tool\n\nThis guide generally assumes that the Ubuntu 22.04 server version (not desktop version) operating system is installed.\n\n## Install miniconda on Ubuntu\n\nA detailed documentation is [here](https://conda.io/projects/conda/en/latest/user-guide/install/index.html). First visit [here](https://conda.io/miniconda.html) to obtain the miniconda downloading path. The following script install conda for all users to the `/opt/miniconda3` directory:\n```sh\nwget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh\nsudo sh Miniconda3-latest-Linux-x86_64.sh -b -p /opt/miniconda3\n```\n\nThen, add conda to the bashrc file so that all users can use the conda command.\n```sh\nsudo vim /etc/bash.bashrc\n# Add the following lines to this file\nexport PATH=\"/opt/miniconda3/bin:$PATH\"\n. /opt/miniconda3/etc/profile.d/conda.sh\n```\n\nAfter that, run the following (or exit the terminal and open a new terminal).\n```sh\nsource /etc/bash.bashrc\n```\n\n## Install miniconda on Mac OS\n\nFor Mac OS, I recommend installing conda by using [Homebrew](https://brew.sh/).\n```sh\nbrew cask install miniconda\necho 'export PATH=\"/usr/local/Caskroom/miniconda/base/bin:$PATH\"' \u003e\u003e ~/.bash_profile\necho '. /usr/local/Caskroom/miniconda/base/etc/profile.d/conda.sh' \u003e\u003e ~/.bash_profile\nsource ~/.bash_profile\n```\n\n## Clone repository and create the conda environment\n\nClone this repository and set the permission.\n```sh\ngit clone --recursive https://github.com/CMU-CREATE-Lab/deep-smoke-machine.git\nsudo chown -R $USER deep-smoke-machine/\nsudo chgrp -R $USER deep-smoke-machine/\nsudo chmod -R 775 deep-smoke-machine/\n```\n\nFor git to ignore permission changes.\n```sh\n# For only this repository\ngit config core.fileMode false\n\n# For globally\ngit config --global core.fileMode false\n```\n\nCreate conda environment and install packages.\n```sh\nconda create -n deep-smoke-machine\nconda activate deep-smoke-machine\nconda install python=3.9\nconda install pip\nwhich pip # make sure this is the pip inside the deep-smoke-machine environment\n```\n\nInstall PyTorch by checking the command on the [PyTorch website](https://pytorch.org). An example for Ubuntu is below:\n```sh\nconda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia\n```\n\nInstall packages.\n```sh\nsh deep-smoke-machine/back-end/install_packages.sh\n```\n\nUpdate the optical_flow submodule.\n```sh\ncd deep-smoke-machine/back-end/www/optical_flow/\ngit submodule update --init --recursive\ngit checkout master\n```\n\nInstall system packages for OpenCV.\n```sh\nsudo apt update\nsudo apt install -y libsm6 libxext6 libxrender-dev\n```\n\n# \u003ca name=\"prepare-data\"\u003e\u003c/a\u003ePrepare data\n\nTo use our publicly released dataset (a snapshot of the [smoke labeling tool](http://smoke.createlab.org/) on 2/24/2020), we include [metadata_02242020.json](back-end/data/dataset/2020-02-24/metadata_02242020.json) file under the `deep-smoke-machine/back-end/data/dataset/` folder. You need to copy, move, and rename this file to `deep-smoke-machine/back-end/data/metadata.json`.\n```sh\ncd deep-smoke-machine/back-end/data/\ncp dataset/2020-02-24/metadata_02242020.json metadata.json\n```\n\nSplit the metadata into three sets: train, validation, and test. This will create a `deep-smoke-machine/back-end/data/split/` folder that contains all splits, as indicated in our paper. The method for splitting the dataset will be explained in the next \"Dataset\" section.\n```sh\ncd deep-smoke-machine/back-end/www/\npython split_metadata.py confirm\n```\n\nDownload all videos in the metadata file to `deep-smoke-machine/back-end/data/videos/`. This will take a very long time, and we recommend running the code on the background using the [screen command](https://www.gnu.org/software/screen/manual/html_node/index.html).\n```sh\npython download_videos.py\n```\n\n\u003e [!IMPORTANT]\n\u003eYou ALWAYS need to deactivate the conda environment before running the `screen` command using `conda deactivate`. When you are inside a screen session, activate your conda environment again to run your code. If you forgot the deactivate conda before you enter a screen session, your code may not run correctly and can give errors.\n\nHere are some tips for the screen command:\n```sh\n# List currently running screen session names\nscreen -ls\n\n# Create a new screen session\nconda deactivate\nscreen\n# Inside the screen, type \"exit\" to terminate the screen\n# Use CTRL+C to interrupt a command\n# Or use CTRL+A+D to detach the screen and send it to the background\n\n# Go into a screen session, below is an example\n# sudo screen -x 33186.download_videos\nscreen -x SCREEN_SESSION_NAME\n\n# Terminate a screen session, below is an example\n# sudo screen -X 33186.download_videos quit\nscreen -X SCREEN_SESSION_NAME quit\n\n# Keep looking at the screen log\ntail -f screenlog.0\n```\n\nProcess and save all videos into RGB frames (under `deep-smoke-machine/back-end/data/rgb/`) and optical flow frames (under `deep-smoke-machine/back-end/data/flow/`). Because computing optical flow takes a very long time, by default, this script will only process RGB frames. If you need the optical flow frames, change the flow_type to 1 in the [`process_videos.py`](back-end/www/process_videos.py) script. The optical flow is only used for the flow-based models in the paper (e.g., `i3d-flow`, `svm-flow`).\n```sh\npython process_videos.py\n```\n\nOptionally, extract [I3D features](https://github.com/piergiaj/pytorch-i3d) under `deep-smoke-machine/back-end/data/i3d_features_rgb/` and `deep-smoke-machine/back-end/data/i3d_features_flow/`. This step is for the SVM models to work in the paper. You do not need to do this if you are not going to use the SVM model (e.g., `svm-rgb`, `svm-flow`).\n```sh\npython extract_features.py METHOD OPTIONAL_MODEL_PATH\n\n# Extract features from pretrained i3d\npython extract_features.py i3d-rgb\npython extract_features.py i3d-flow\n\n# Extract features from a saved i3d model\npython extract_features.py i3d-rgb ../data/saved_i3d/ecf7308-i3d-rgb/model/16875.pt\npython extract_features.py i3d-flow ../data/saved_i3d/af00751-i3d-flow/model/30060.pt\n```\n\n# \u003ca name=\"train-test-models\"\u003e\u003c/a\u003eTrain and test models\n\nTrain the model with cross-validation on all dataset splits, using different hyper-parameters. The model will be trained on the training set and validated on the validation set. Pretrained weights are obtained from the [pytorch-i3d repository](https://github.com/piergiaj/pytorch-i3d). By default, the information of the trained I3D model will be placed in the `deep-smoke-machine/back-end/data/saved_i3d/` folder. For the description of the models, please refer to our paper. Note that by default the PyTorch [DistributedDataParallel](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) GPU parallel computing is enabled (see [i3d_learner.py](back-end/www/i3d_learner.py)).\n```sh\npython train.py METHOD OPTIONAL_MODEL_PATH\n\n# Use Two-Stream Inflated 3D ConvNet\npython train.py i3d-rgb-cv-1\n\n# Use I3D features + SVM\npython train.py svm-rgb-cv-1\n```\n\n\u003e [!TIP]\n\u003e When training models, you can use `watch -n 1 nvidia-smi` to monitor GPU usage, such as checking if you run out of GPU memory.\n\nTest the performance of a model on the test set. This step will also generate summary videos for each cell in the confusion matrix (true positive, true negative, false positive, and false negative).\n```sh\npython test.py METHOD OPTIONAL_MODEL_PATH\n\n# Use Two-Stream Inflated 3D ConvNet\npython test.py i3d-rgb-cv-1 ../data/saved_i3d/ecf7308-i3d-rgb/model/16875.pt\n\n# Use I3D features + SVM\npython test.py svm-rgb-cv-1 ../data/saved_svm/445cc62-svm-rgb/model/model.pkl\n```\n\nRun [Grad-CAM](https://arxiv.org/abs/1610.02391) to visualize the areas in the videos that the model is looking at.\n```sh\npython grad_cam_viz.py i3d-rgb MODEL_PATH\n```\n\nAfter model training and testing, the folder structure will look like the following:\n```sh\n└── saved_i3d                            # this corresponds to deep-smoke-machine/back-end/data/saved_i3d/\n    └── 549f8df-i3d-rgb-s1               # the name of the model, s1 means split 1\n        ├── cam                          # the visualization using Grad-CAM\n        ├── log                          # the log when training models\n        ├── metadata                     # the metadata of the dataset split\n        ├── model                        # the saved models\n        ├── run                          # the saved information for TensorBoard\n        └── viz                          # the sampled videos for each cell in the confusion matrix\n```\n\nIf you want to see the training and testing results on [TensorBoard](https://pytorch.org/docs/stable/tensorboard.html), run the following and go to the stated URL in your browser.\n```sh\ncd deep-smoke-machine/back-end/data/\ntensorboard --logdir=saved_i3d\n```\n\nRecommended training strategy:\n1. Set an initial learning rate (e.g., 0.1)\n2. Keep this learning rate and train the model until the training error decreases too slow (or fluctuate) or until the validation error increases (a sign of overfitting)\n3. Decrease the learning rate (e.g., by a factor of 10)\n4. Load the best model weight from the ones that were trained using the previous learning rate\n5. Repeat step 2, 3, and 4 until convergence\n\n# \u003ca name=\"code-structure\"\u003e\u003c/a\u003eCode infrastructure\n\nThis section explains the code infrastructure related to the I3D model training and testing in the [deep-smoke-machine/back-end/www/](back-end/www/) folder. Later in this section, I will describe how to build your own model and integrate it with the current pipeline. This code assumes that you are familiar with the [PyTorch deep learning framework](https://pytorch.org/). If you do not know PyTorch, I recommend checking [their tutorial page](https://pytorch.org/tutorials/) first.\n- [base_learner.py](back-end/www/base_learner.py)\n  - The abstract class for creating model learners. You will need to implement the fit and test function. This script provides shared functions, such as model loading, model saving, data augmentation, and progress logging.\n- [i3d_learner.py](back-end/www/i3d_learner.py)\n  - This script inherits the base_learner.py script for training the I3D models. This script contains code for back-propagation (e.g., loss function, learning rate scheduler, video batch loading) and GPU parallel computing (PyTorch `DistributedDataParallel`).\n- [check_models.py](back-end/www/check_models.py)\n  - Check if a developed model runs in simple cases. This script is used for debugging when developing new models.\n- [smoke_video_dataset.py](back-end/www/smoke_video_dataset.py)\n  - Definition of the dataset. This script inherits the PyTorch Dataset class for creating the DataLoader, which can be used to provide batches iteratively when training the models.\n- [opencv_functional.py](back-end/www/opencv_functional.py)\n  - A special utility function that mimics [torchvision.transforms.functional](https://pytorch.org/docs/stable/_modules/torchvision/transforms/functional.html), designed for processing video frames and augmenting video data.\n- [video_transforms.py](back-end/www/video_transforms.py)\n  - A special utility function that mimics [torchvision.transforms.transforms](https://pytorch.org/docs/stable/_modules/torchvision/transforms/transforms.html), designed for processing video frames and augmenting video data.\n- [deep-smoke-machine/back-end/www/model/](back-end/www/model/)\n  - The place to put all models (e.g., I3D, Non-Local modules, Timeception modules, Temporal Shift modules, LSTM).\n\nIf you want to develop your own model, here are the steps that I recommend.\n1. Play with the check_models.py script to understand the input and output dimensions.\n2. Create your own model and place it in the `deep-smoke-machine/back-end/www/model/` folder. You can take a look at other models to get an idea about how to write the code.\n3. Import your model to the `check_models.py` script, then run the script to debug your model.\n4. If you need a specific data augmentation pipeline, edit the get_transform function in the `base_learner.py` file. Depending on your needs, you may also need to edit the `opencv_functional.py` and `video_transforms.py` files.\n5. Copy the `i3d_learner.py` file, import your model, and modify the code to suit your needs. Make sure that you import your customized learner class in the `train.py` and `test.py` files.\n\n# \u003ca name=\"dataset\"\u003e\u003c/a\u003eDataset\n\nWe include our publicly released dataset (a snapshot of the [smoke labeling tool](http://smoke.createlab.org/) on 2/24/2020) [metadata_02242020.json](back-end/data/dataset/2020-02-24/metadata_02242020.json) file under the `deep-smoke-machine/back-end/data/dataset/` folder. The JSON file contains an array, with each element in the array representing the metadata for a video. Each element is a dictionary with keys and values, explained below:\n- camera_id\n  - ID of the camera (0 means [clairton1](http://mon.createlab.org/#v=3703.5,970,0.61,pts\u0026t=456.42\u0026ps=25\u0026d=2020-04-06\u0026s=clairton1\u0026bt=20200406\u0026et=20200407), 1 means [braddock1](http://mon.createlab.org/#v=2868.5,740.5,0.61,pts\u0026t=540.67\u0026ps=25\u0026d=2020-04-07\u0026s=braddock1\u0026bt=20200407\u0026et=20200408), and 2 means [westmifflin1](http://mon.createlab.org/#v=1722.89321,1348.42994,0.806,pts\u0026t=704.33\u0026ps=25\u0026d=2020-04-07\u0026s=westmifflin1\u0026bt=20200407\u0026et=20200408))\n- view_id\n  - ID of the cropped view from the camera\n  - Each camera produces a panarama, and each view is cropped from this panarama (will be explained later in this section)\n- id\n  - Unique ID of the video clip\n- label_state\n  - State of the video label produced by the citizen science volunteers (will be explained later in this section)\n- label_state_admin\n  - State of the video label produced by the researchers (will be explained later in this section)\n- start_time\n  - Starting epoch time (in seconds) when capturing the video, corresponding to the real-world time\n- url_root\n  - URL root of the video, need to combine with url_part to get the full URL (url_root + url_part)\n- url_part\n  - URL part of the video, need to combine with url_root to get the full URL (url_root + url_part)\n- file_name\n  - File name of the video, for example 0-1-2018-12-13-6007-928-6509-1430-180-180-6614-1544720610-1544720785\n  - The format of the file_name is [camera_id]-[view_id]-[year]-[month]-[day]-[bound_left]-[bound_top]-[bound_right]-[bound_bottom]-[video_height]-[video_width]-[start_frame_number]-[start_epoch_time]-[end_epoch_time]\n  - bound_left, bound_top, bound_right, and bound_bottom mean the bounding box of the video clip in the panarama\n\nNote that the url_root and url_part point to videos with 180 by 180 resolutions. We also provide a higher resolution (320 by 320) version of the videos. Replace the \"/180/\" with \"/320/\" in the url_root, and also replace the \"-180-180-\" with \"-320-320-\" in the url_part. For example, see the following:\n- URL for the 180 by 180 version: https://smoke.createlab.org/videos/180/2019-06-24/0-7/0-7-2019-06-24-3504-1067-4125-1688-180-180-9722-1561410615-1561410790.mp4\n- URL for the 320 by 320 version: https://smoke.createlab.org/videos/320/2019-06-24/0-7/0-7-2019-06-24-3504-1067-4125-1688-320-320-9722-1561410615-1561410790.mp4\n\nEach video is reviewed by at least two citizen science volunteers (or one researcher who received the [smoke reading training](https://www.eta-is-opacity.com/resources/method-9/)). Our paper describes the details of the labeling and quality control mechanism. The state of the label (label_state and label_state_admin) in the `metadata_02242020.json` is briefly explained below.\n- 47 : gold standard positive\n  - The researcher assigned a positive label to the video and indicated that the video should be used as a gold standard for data quality checks.\n- 32 : gold standard negative\n  - The researcher assigned a negative label to the video and indicated that the video should be used as a gold standard for data quality checks.\n- 23 : strong positive\n  - Two volunteers both agree (or one researcher says) that the video has smoke.\n- 16 : strong negative\n  - Two volunteers both agree (or one researcher says) that the video does not have smoke.\n- 19 : weak positive\n  - Two volunteers have different answers, and the third volunteer says that the video has smoke.\n- 20 : weak negative\n  - Two volunteers have different answers, and the third volunteer says that the video does not have smoke.\n- 5 : maybe positive\n  - One volunteers says that the video has smoke.\n- 4 : maybe negative\n  - One volunteers says that the video does not have smoke.\n- 3 : has discord\n  - Two volunteers have different answers (one says yes, and another one says no).\n- -1 : no data, no discord\n  - No data. If label_state_admin is -1, it means that the label is produced solely by citizen science volunteers. If label_state is -1, it means that the label is produced solely by researchers. Otherwise, the label is jointly produced by both citizen science volunteers and researchers. Please refer to our paper about these three cases.\n- -2 : bad videos\n  - This means that reseachers have checked the data and marked the video as not suitable for labeling (e.g., due to bad data quality such as incorrect image stitching or artifacts during video compression). These bad videos should not be used in building the model.\n\nAfter running the [split_metadata.py](back-end/www/split_metadata.py) script, the \"label_state\" and \"label_state_admin\" keys in the dictionary will be aggregated into the final label, represented by the new \"label\" key (see the JSON files in the generated `deep-smoke-machine/back-end/data/split/` folder). Positive (value 1) and negative (value 0) labels mean if the video clip has smoke emissions or not, respectively.\n\nAlso, the dataset will be divided into several splits, based on camera views or dates. The file names (without \".json\" file extension) are listed below. The Split S\u003csub\u003e0\u003c/sub\u003e, S\u003csub\u003e1\u003c/sub\u003e, S\u003csub\u003e2\u003c/sub\u003e, S\u003csub\u003e3\u003c/sub\u003e, S\u003csub\u003e4\u003c/sub\u003e, and S\u003csub\u003e5\u003c/sub\u003e correspond to the ones indicated in the paper.\n\n| Split | Train | Validate | Test |\n| --- | --- | --- | --- |\n| S\u003csub\u003e0\u003c/sub\u003e | metadata_train_split_0_by_camera | metadata_validation_split_0_by_camera | metadata_test_split_0_by_camera |\n| S\u003csub\u003e1\u003c/sub\u003e | metadata_train_split_1_by_camera | metadata_validation_split_1_by_camera | metadata_test_split_1_by_camera |\n| S\u003csub\u003e2\u003c/sub\u003e | metadata_train_split_2_by_camera | metadata_validation_split_2_by_camera | metadata_test_split_2_by_camera |\n| S\u003csub\u003e3\u003c/sub\u003e | metadata_train_split_by_date | metadata_validation_split_by_date | metadata_test_split_by_date |\n| S\u003csub\u003e4\u003c/sub\u003e | metadata_train_split_3_by_camera | metadata_validation_split_3_by_camera | metadata_test_split_3_by_camera |\n| S\u003csub\u003e5\u003c/sub\u003e | metadata_train_split_4_by_camera | metadata_validation_split_4_by_camera | metadata_test_split_4_by_camera |\n\nThe following table shows the content in each split, except S\u003csub\u003e3\u003c/sub\u003e. The splitting strategy is that each view will be present in the testing set at least once. Also, the camera views that monitor different industrial facilities (view 1-0, 2-0, 2-1, and 2-2) are always on the testing set. Examples of the camera views will be provided later in this section.\n\n| View | S\u003csub\u003e0\u003c/sub\u003e | S\u003csub\u003e1\u003c/sub\u003e | S\u003csub\u003e2\u003c/sub\u003e | S\u003csub\u003e4\u003c/sub\u003e | S\u003csub\u003e5\u003c/sub\u003e |\n| --- | --- | --- | --- | --- | --- |\n| 0-0 | Train | Train | Test | Train | Train |\n| 0-1 | Test | Train | Train | Train | Train |\n| 0-2 | Train | Test | Train | Train | Train |\n| 0-3 | Train | Train | Validate | Train | Test |\n| 0-4 | Validate | Train | Train | Test | Validate |\n| 0-5 | Train | Validate | Train | Train | Test |\n| 0-6 | Train | Train | Test | Train | Validate |\n| 0-7 | Test | Train | Train | Validate | Train |\n| 0-8 | Train | Train | Validate | Test | Train |\n| 0-9 | Train | Test | Train | Validate | Train |\n| 0-10 | Validate | Train | Train | Test | Train |\n| 0-11 | Train | Validate | Train | Train | Test |\n| 0-12 | Train | Train | Test | Train | Train |\n| 0-13 | Test | Train | Train | Train | Train |\n| 0-14 | Train | Test | Train | Train | Train |\n| 1-0 | Test | Test | Test | Test | Test |\n| 2-0 | Test | Test | Test | Test | Test |\n| 2-1 | Test | Test | Test | Test | Test |\n| 2-2 | Test | Test | Test | Test | Test |\n\nThe following shows the split of S\u003csub\u003e3\u003c/sub\u003e by time sequence, where the farthermost 18 days are used for training, the middle 2 days are used for validation, and the nearest 10 days are used for testing. You can find our camera data by date on [our air pollution monitoring network](http://mon.createlab.org/).\n- Training set of S\u003csub\u003e3\u003c/sub\u003e\n  - 2018-05-11, 2018-06-11, 2018-06-12, 2018-06-14, 2018-07-07, 2018-08-06, 2018-08-24, 2018-09-03, 2018-09-19, 2018-10-07, 2018-11-10, 2018-11-12, 2018-12-11, 2018-12-13, 2018-12-28, 2019-01-11, 2019-01-17, 2019-01-18\n- Validation set of S\u003csub\u003e3\u003c/sub\u003e\n  - 2019-01-22, 2019-02-02\n- Testing set of S\u003csub\u003e3\u003c/sub\u003e\n  - 2019-02-03, 2019-02-04, 2019-03-14, 2019-04-01, 2019-04-07, 2019-04-09, 2019-05-15, 2019-06-24, 2019-07-26, 2019-08-11\n\nThe dataset contains 12,567 clips with 19 distinct views from cameras on three sites that monitored three different industrial facilities. The clips are from 30 days that spans four seasons in two years in the daytime. The following provides examples and the distribution of labels for each camera view, with the format [camera_id]-[view_id]:\n\n![This figure shows a part of the dataset.](back-end/data/dataset/2020-02-24/dataset_1.png)\n\n![This figure shows a part of the dataset.](back-end/data/dataset/2020-02-24/dataset_2.png)\n\n![This figure shows a part of the dataset.](back-end/data/dataset/2020-02-24/dataset_3.png)\n\n![This figure shows a part of the dataset.](back-end/data/dataset/2020-02-24/dataset_4.png)\n\nWe made sure that we were not invading the privacy of surrounding residential neighbors. Areas in the videos that look inside house windows were cropped or blocked. Also, there is no law in our region to prohibit the monitoring of industrial activities.\n\n# \u003ca name=\"pretrained-models\"\u003e\u003c/a\u003ePretrained models\n\nWe release two of our best baseline models: [RGB-I3D](back-end/data/pretrained_models/RGB-I3D-S3.pt) and [RGB-TC](https://github.com/CMU-CREATE-Lab/deep-smoke-machine/blob/master/back-end/data/pretrained_models/RGB-TC-S3.pt), both trained and tested on split S\u003csub\u003e3\u003c/sub\u003e using four NVIDIA GTX 1080 Ti GPUs. Please feel free to finetune your models based on our baseline. Our paper describes the details of these models. RGB-I3D uses [I3D ConvNet architecture with Inception-v1 layers](https://arxiv.org/pdf/1705.07750.pdf) and RGB frame input. RGB-TC is finetuned from RGB-I3D, with additional [Timeception](https://arxiv.org/pdf/1812.01289.pdf) layers. Below shows an example usage:\n```python\n# Import I3D\nfrom i3d_learner import I3dLearner\n\n# Initialize the model\nmodel = I3dLearner(\n    mode=\"rgb\",\n    augment=True,\n    p_frame=\"../data/rgb/\",\n    use_tc=True,\n    freeze_i3d=True,\n    batch_size_train=8,\n    milestones=[1000, 2000],\n    num_steps_per_update=1)\n\n# Finetune the RGB-TC model from the RGB-I3D model\nmodel.fit(\n    p_model=\"../data/pretrained_models/RGB-I3D-S3.pt\",\n    model_id_suffix=\"-s3\",\n    p_metadata_train=\"../data/split/metadata_train_split_by_date.json\",\n    p_metadata_validation=\"../data/split/metadata_validation_split_by_date.json\",\n    p_metadata_test=\"../data/split/metadata_test_split_by_date.json\")\n```\n\nBesides these two model weights that are included in this repository, we also released the model weights for all the splits for model RGB-I3D and RGB-TC on [a Figshare repository](https://doi.org/10.21942/uva.25705239.v1). To use these weights, first download and extract the zip file. Then, place the extracted `paper_result` folder under the `deep-smoke-machine/back-end/data` directory. The code below is an example:\n```python\n# Import I3D\nfrom i3d_learner import I3dLearner\n\n# Initialize the model\nmodel = I3dLearner(\n    mode=\"rgb\",\n    augment=True,\n    p_frame=\"../data/rgb/\",\n    use_tc=True,\n    freeze_i3d=True,\n    batch_size_train=8,\n    milestones=[1000, 2000],\n    num_steps_per_update=1)\n\n# Change this to your split number (can be \"s0\", \"s1\", \"s2\", \"s3\", \"s4\", or \"s5\")\nsplit_str = \"s0\"\n\n# Make sure that the \"paper_result\" folder is in the \"data\" foler\nmodel_root_path = \"../data/paper_result/RGB-I3D/\" + split_str\n\n# Change this to the correct model number under the \"model\" folder\np_model = model_root_path + \"/model/682.pt\"\n\n# No need to change the variables below\nmodel_id_suffix = \"-\" + split_str\np_metadata_train = model_root_path + \"/metadata/metadata_train.json\"\np_metadata_validation = model_root_path + \"/metadata/metadata_validation.json\"\np_metadata_test = model_root_path + \"/metadata/metadata_test.json\"\n\n# Finetune the RGB-TC model from the RGB-I3D model\nmodel.fit(\n    p_model=p_model,\n    model_id_suffix=model_id_suffix,\n    p_metadata_train=p_metadata_train,\n    p_metadata_validation=p_metadata_validation,\n    p_metadata_test=p_metadata_test)\n```\n\n# \u003ca name=\"deploy-models-to-recognize-smoke\"\u003e\u003c/a\u003eDeploy models to recognize smoke\n\nWe provide an example script ([recognize_smoke.py](back-end/www/recognize_smoke.py)) to show how you can deploy the trained models to recognize industrial smoke emissions. This script only works for the videos on our camera monitoring system ([http://mon.createlab.org/](http://mon.createlab.org/)) or others that are created using the [timemachine-creator](https://github.com/CMU-CREATE-Lab/timemachine-creator) and [timemachine-viewer](https://github.com/CMU-CREATE-Lab/timemachine-viewer). In sum, the script takes a list of video URLs (examples can be found [here](https://github.com/CMU-CREATE-Lab/deep-smoke-machine/blob/improve-documentation/back-end/data/production_url_list/2019-01-03.json)), gets their date and camera view boundary information, generates a bunch of cropped clips, and run the model on these clips to recognize smoke emissions. Here are the steps:\n\nFirst, for a date that you want to process, create a JSON file under the [back-end/data/production_url_list/](back-end/data/production_url_list/) folder to add video URLs. The format of the file name must be \"YYYY-MM-DD.json\" (such as \"2019-01-03.json\"). If the file for that date exists, just open the file and add more video URLs. Each video URL is specified using a dictionary, and you need to put the video URLs in a list in each JSON file. For example:\n```json\n[{\n  \"url\": \"https://thumbnails-v2.createlab.org/thumbnail?root=https://tiles.cmucreatelab.org/ecam/timemachines/clairton1/2019-01-03.timemachine/\u0026width=180\u0026height=180\u0026startFrame=9716\u0026format=mp4\u0026fps=12\u0026tileFormat=mp4\u0026startDwell=0\u0026endDwell=0\u0026boundsLTRB=6304,884,6807,1387\u0026nframes=36\",\n  \"cam_id\": 0,\n  \"view_id\": 0\n  },{\n  \"url\": \"https://thumbnails-v2.createlab.org/thumbnail?root=https://tiles.cmucreatelab.org/ecam/timemachines/clairton1/2019-01-03.timemachine/\u0026width=180\u0026height=180\u0026startFrame=9716\u0026format=mp4\u0026fps=12\u0026tileFormat=mp4\u0026startDwell=0\u0026endDwell=0\u0026boundsLTRB=6007,928,6509,1430\u0026nframes=36\",\n  \"cam_id\": 0,\n  \"view_id\": 1\n}]\n```\n\nThe URL indicates the cropped video clips, which is obtained by using the thumbnail tool on our camera monitoring system ([http://mon.createlab.org/](http://mon.createlab.org/)). To access the thumbnail tool, click the \"share\" button at the bottom-right near the timeline slider and then select the \"Share as image or video\" tab. A tutorial about how to use the thumbnail tool for sharing videos can be found [here](https://vimeo.com/140196813#t=415s). The cam_id and view_id correspond to the camera views presented in the \"Dataset\" section in this READEME. For example, if cam_id is 0 and view_id is 1, this means that the camera view is 0-1, as shown in [this graph](back-end/data/dataset/2020-02-24/dataset_1.png). After creating the JSON files or adding video URLs to existing JSON files, run the following to perform a sanity check, which will identify problems related to the camera data and attemp to fix the problems:\n```sh\npython recognize_smoke.py check_and_fix_urls\n```\n\nNext, run the following (this step will take a long time) to process each clip and predict the probability of having smoke:\n```sh\npython recognize_smoke.py process_all_urls\n```\n\nThis will create a \"production\" folder under [back-end/data/](back-end/data) to store the processed results. Then, run the following to identify events based on the probabilities of having smoke:\n```sh\npython recognize_smoke.py process_events\n```\n\nThis will create an \"event\" folder under [back-end/data/](back-end/data) to store the links to the video clips that are identified as having smoke emissions. To visualize the smoke events, copy the folder (with the same folder name, \"event\") to the front-end of the [video labeling tool](https://github.com/CMU-CREATE-Lab/video-labeling-tool/tree/master/front-end). Then, the [event page](https://smoke.createlab.org/event.html?date=2019-04-02\u0026camera=0\u0026view=all) will be able to access the \"event\" folder and show the results. You may also want to consider running the following to scan the video clips so that users do not need to wait for the [thumbnail server](https://thumbnails-v2.createlab.org/status) to render videos:\n```sh\npython recognize_smoke.py scan_urls\n```\n\n# \u003ca name=\"admin\"\u003e\u003c/a\u003eAdministrator only tasks\n\nFor researchers in our team, if you wish to update the dataset, you need to obtain user token from the [smoke labeling tool](https://smoke.createlab.org/gallery.html) and put the `user_token.js` file in the `deep-smoke-machine/back-end/data/` directory. You need permissions from the system administrator of the smoke labeling tool to download the user token. After getting the token, get the video metadata (using the command below). This will create a `metadata.json` file under `deep-smoke-machine/back-end/data/`.\n```sh\n# This is only for system\npython get_metadata.py confirm\n```\n\n# \u003ca name=\"acknowledgements\"\u003e\u003c/a\u003eAcknowledgements\n\nWe thank [GASP](https://gasp-pgh.org/) (Group Against Smog and Pollution), [Clean Air Council](https://cleanair.org/), [ACCAN](https://accan.org/) (Allegheny County Clean Air Now), [Breathe Project](https://breatheproject.org/), [NVIDIA](https://developer.nvidia.com/academic_gpu_seeding), and the [Heinz Endowments](http://www.heinz.org/) for the support of this research. We also greatly appreciate the help of our volunteers, which includes labeling videos and providing feedback in system development.\n","funding_links":[],"readme_doi_urls":["https://doi.org/10.21942/uva.25705239.v1"],"works":{},"citation_counts":{},"total_citations":0,"keywords_from_contributors":[],"project_url":"https://ost.ecosyste.ms/api/v1/projects/20235","html_url":"https://ost.ecosyste.ms/projects/20235"}