BioCLIP

A foundation model for the tree of life, leveraging the unique properties of biology captured by TreeOfLife-10M, namely the abundance and variety of images of plants, animals, and fungi, together with the availability of rich structured biological knowledge.
https://github.com/imageomics/bioclip

Category: Biosphere
Sub Category: Biodiversity Data Access and Management

Keywords

clip computer-vision imageomics knowledge-guided-machine-learning taxonomy

Keywords from Contributors

transformers language-model jax bert speech pretrained-models charting benchmarking resnet tensor

Last synced: about 8 hours ago
JSON representation

Repository metadata

This is the repository for the BioCLIP model and the TreeOfLife-10M dataset [CVPR'24 Oral, Best Student Paper].

• Host: GitHub
• URL: https://github.com/imageomics/bioclip
• Owner: Imageomics
• License: other
• Created: 2023-11-30T15:40:33.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2025-02-26T17:01:30.000Z (about 2 months ago)
• Last Synced: 2025-04-20T04:03:02.236Z (7 days ago)
• Topics: clip, computer-vision, imageomics, knowledge-guided-machine-learning, taxonomy
• Language: Python
• Homepage: https://imageomics.github.io/bioclip/
• Size: 28.9 MB
• Stars: 199
• Watchers: 15
• Forks: 18
• Open Issues: 2
• Releases: 3
• Metadata Files:
  • Readme: README.md
  • Changelog: HISTORY.md
  • License: LICENSE
  • Citation: CITATION.cff

README.md

@inproceedings{stevens2024bioclip,
  title = {{B}io{CLIP}: A Vision Foundation Model for the Tree of Life}, 
  author = {Samuel Stevens and Jiaman Wu and Matthew J Thompson and Elizabeth G Campolongo and Chan Hee Song and David Edward Carlyn and Li Dong and Wasila M Dahdul and Charles Stewart and Tanya Berger-Wolf and Wei-Lun Chao and Yu Su},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2024},
  pages = {19412-19424}
}

@software{bioclip2023code,
  author = {Samuel Stevens and Jiaman Wu and Matthew J. Thompson and Elizabeth G. Campolongo and Chan Hee Song and David Edward Carlyn},
  doi = {10.5281/zenodo.10895871},
  title = {BioCLIP},
  version = {v1.0.0},
  year = {2024}
}

@software{ilharco_gabriel_2021_5143773,
  author={Ilharco, Gabriel and Wortsman, Mitchell and Wightman, Ross and Gordon, Cade and Carlini, Nicholas and Taori, Rohan and Dave, Achal and Shankar, Vaishaal and Namkoong, Hongseok and Miller, John and Hajishirzi, Hannaneh and Farhadi, Ali and Schmidt, Ludwig},
  title={OpenCLIP},
  year={2021},
  doi={10.5281/zenodo.5143773},
}

@misc{inat2021,
  author={Van Horn, Grant and Mac Aodha, Oisin},
  title={iNat Challenge 2021 - FGVC8},
  publisher={Kaggle},
  year={2021},
  url={https://kaggle.com/competitions/inaturalist-2021}
}

@inproceedings{gharaee2023step,
  author={Gharaee, Z. and Gong, Z. and Pellegrino, N. and Zarubiieva, I. and Haurum, J. B. and Lowe, S. C. and McKeown, J. T. A. and Ho, C. Y. and McLeod, J. and Wei, Y. C. and Agda, J. and Ratnasingham, S. and Steinke, D. and Chang, A. X. and Taylor, G. W. and Fieguth, P.},
  title={A Step Towards Worldwide Biodiversity Assessment: The {BIOSCAN-1M} Insect Dataset},
  booktitle={Advances in Neural Information Processing Systems ({NeurIPS}) Datasets \& Benchmarks Track},
  year={2023},
}

Citation (CITATION.cff)

---
abstract: "Images of the natural world are an abundant source of biological
  information. There are many computational methods and tools, particularly
  computer vision, for extracting information from images. However, existing
  methods consist of bespoke models, not adaptable or extendable from their
  targeted task to new questions, contexts, and datasets. We thus develop the
  first large-scale multimodal model, BioCLIP, for general biology questions on
  images. We leverage the unique properties of biology (abundance and variety of
  images and availability of rich structured biological knowledge) as the the
  application domain for computer vision."
authors:
  - family-names: Stevens
    given-names: Samuel
  - family-names: Wu
    given-names: Jiaman
  - family-names: Thompson
    given-names: "Matthew J."
  - family-names: Campolongo
    given-names: "Elizabeth G."
  - family-names: Song
    given-names: "Chan Hee"
  - family-names: Carlyn
    given-names: "David Edward"
cff-version: 1.2.0
date-released: "2024-09-19"
identifiers:
  - doi: "10.5281/zenodo.10895870"
  - description: "The GitHub release URL of tag v1.0.2."
    type: url
    value: "https://github.com/Imageomics/bioclip/releases/tag/v1.0.2"
  - description: "The GitHub URL of the commit tagged with v1.0.2."
    type: url
    value: "https://github.com/Imageomics/bioclip/tree/b750fa8758d16e78ccc3de7ab86c1523d5de6148"
keywords:
  - clip
  - biology
  - CV
  - imageomics
  - animals
  - species
  - images
  - taxonomy
  - "rare species"
  - "endangered species"
  - "evolutionary biology"
  - multimodal
  - "knowledge-guided"
license: MIT
message: "If you use this software, please cite both the article from preferred-citation and the software itself."
repository-code: "https://github.com/Imageomics/bioclip"
title: BioCLIP
version: 1.0.2
type: software
preferred-citation:
  type: conference-paper
  authors:
    - family-names: Stevens
      given-names: Samuel
    - family-names: Wu
      given-names: Jiaman
    - family-names: Thompson
      given-names: "Matthew J."
    - family-names: Campolongo
      given-names: "Elizabeth G."
    - family-names: Song
      given-names: "Chan Hee"
    - family-names: Carlyn
      given-names: "David Edward"
    - family-names: Dong
      given-names: Li
    - family-names: Dahdul
      given-names: "Wasila M"
    - family-names: Stewart
      given-names: Charles
    - family-names: "Berger-Wolf"
      given-names: Tanya
    - family-names: Chao
      given-names: "Wei-Lun"
    - family-names: Su
      given-names: Yu
  title: "BioCLIP: A Vision Foundation Model for the Tree of Life"
  year: 2024
  pages: "19412-19424"
  collection-title: "Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)"
references:
  - authors:
      - family-names: Ilharco
        given-names: Gabriel
      - family-names: Wortsman
        given-names: Mitchell
      - family-names: Wightman
        given-names: Ross
      - family-names: Gordon
        given-names: Cade
      - family-names: Carlini
        given-names: Nicholas
      - family-names: Taori
        given-names: Rohan
      - family-names: Dave
        given-names: Achal
      - family-names: Shankar
        given-names: Vaishaal
      - family-names: Namkoong
        given-names: Hongseok
      - family-names: Miller
        given-names: John
      - family-names: Hajishirzi
        given-names: Hannaneh
      - family-names: Farhadi
        given-names: Ali
      - family-names: Schmidt
        given-names: Ludwig
    title: OpenCLIP
    version: v0.1
    type: software
    doi: "10.5281/zenodo.5143773"
    date-released: "2021-07-28"

Owner metadata

• Name: Imageomics Institute
• Login: Imageomics
• Email:
• Kind: organization
• Description:
• Website: https://imageomics.osu.edu
• Location:
• Twitter: imageomics
• Company:
• Icon url: https://avatars.githubusercontent.com/u/77468321?v=4
• Repositories: 4
• Last ynced at: 2023-03-05T05:05:36.574Z
• Profile URL: https://github.com/Imageomics

GitHub Events

Total

• Issues event: 5
• Watch event: 42
• Delete event: 1
• Issue comment event: 4
• Push event: 1
• Pull request review event: 1
• Pull request event: 3
• Fork event: 6
• Create event: 1

Last Year

• Issues event: 5
• Watch event: 42
• Delete event: 1
• Issue comment event: 4
• Push event: 1
• Pull request review event: 1
• Pull request event: 3
• Fork event: 6
• Create event: 1

Committers metadata

Last synced: 7 days ago

Committer domains:

• me.com: 1
• keke.dev: 1
• umich.edu: 1
• berkeley.edu: 1
• ip-26-0-132-138.ec2.internal: 1
• ip-26-0-162-192.ec2.internal: 1
• ip-172-31-34-110.ec2.internal: 1
• reseau.eseo.fr: 1
• carlini.com: 1
• fz-juelich.de: 1
• osu.edu: 1

Issue and Pull Request metadata

Last synced: 1 day ago

Dependencies

Score: 8.992184362173012