BIOSCAN-5M

Overview

This repository contains the code and data related to the BIOSCAN-5M
project.
BIOSCAN-5M is a comprehensive multi-modal dataset comprised of over 5 million specimens, 98% of which are insects.
Every record has both image and DNA data.

If you make use of the BIOSCAN-5M dataset and/or this code repository, please cite the following paper:

@inproceedings{gharaee2024bioscan5m,
    title={{BIOSCAN-5M}: A Multimodal Dataset for Insect Biodiversity},
    booktitle={Advances in Neural Information Processing Systems},
    author={Zahra Gharaee and Scott C. Lowe and ZeMing Gong and Pablo Millan Arias
        and Nicholas Pellegrino and Austin T. Wang and Joakim Bruslund Haurum
        and Iuliia Zarubiieva and Lila Kari and Dirk Steinke and Graham W. Taylor
        and Paul Fieguth and Angel X. Chang
    },
    editor={A. Globerson and L. Mackey and D. Belgrave and A. Fan and U. Paquet and J. Tomczak and C. Zhang},
    pages={36285--36313},
    publisher={Curran Associates, Inc.},
    year={2024},
    volume={37},
    url={https://proceedings.neurips.cc/paper_files/paper/2024/file/3fdbb472813041c9ecef04c20c2b1e5a-Paper-Datasets_and_Benchmarks_Track.pdf},
}

Getting Started with BIOSCAN-5M

I. Environment Setup

To set up the BIOSCAN-5M project, create the required environment using the provided bioscan5m.yaml file. Run the following command:

conda env create -f bioscan5m.yaml

II. Dataset Quick Start

Quickly access the BIOSCAN-5M dataset by installing the dataset package and initializing the data loader. Use the following commands:

pip install bioscan-dataset

from bioscan_dataset import BIOSCAN5M

ds = BIOSCAN5M("~/Datasets/bioscan-5m", download=True)

For more detailed information, please visit BIOSCAN-5M Dataset Package

III. Task-Specific Settings

Please note that to work with all modules connected to this repository,
you may need to install additional dependencies specific to each module (if any).
Be sure to follow the instructions provided within each module's folder for further setup details.

Dataset

We present BIOSCAN-5M dataset to the machine learning community.
We hope this dataset will facilitate the development of tools to automate aspects of the monitoring of global insect biodiversity.

Each record of the BIOSCAN-5M dataset contains six primary attributes:

• RGB image
  • Metadata field: processid
• DNA barcode sequence
  • Metadata field: dna_barcode
• Barcode Index Number (BIN)
  • Metadata field: dna_bin
• Biological taxonomic classification
  • Metadata fields: phylum, class, order, family, subfamily, genus, species, taxon
• Geographical information
  • Metadata fields: country, province_state, latitude, longitude
• Specimen size
  • Metadata fields: image_measurement_value, area_fraction, scale_factor

Dataset Access

All dataset image packages and metadata files are accessible for download through the
GoogleDrive folder.
Additionally, the dataset is available on research and data sharing platforms such as Zenodo,
Kaggle, and HuggingFace.

Dataset Browser

The BIOSCAN-5M Dataset Browser is an interactive tool designed to explore the BIOSCAN-5M dataset efficiently.
It allows you to navigate through taxonomic ranks, visualize specimens, and analyze DNA barcode sequences.
The browser supports advanced filtering, sorting, and visualization capabilities to facilitate in-depth data exploration for researchers and developers.

Metadata

The dataset metadata file BIOSCAN_5M_Insect_Dataset_metadata contains biological information, geographic information as well as
size information of the organisms. We provide this metadata in both CSV and JSONLD file types.

RGB Image

The BIOSCAN-5M dataset comprises resized and cropped images.
We have provided various packages of the BIOSCAN-5M dataset, each tailored for specific purposes.

Cropped images

We trained a model on examples from this dataset in order to create a tool introduced in BIOSCAN-1M, which can automatically generate bounding boxes around the insect.
We used this to crop each image down to only the region of interest.

Image packages

• BIOSCAN_5M_original_full: The raw images of the dataset.
• BIOSCAN_5M_cropped: Images after cropping with our cropping tool.
• BIOSCAN_5M_original_256: Original images resized to 256 on their shorter side.
• BIOSCAN_5M_cropped_256: Cropped images resized to 256 on their shorter side.

Geographical Information

The BIOSCAN-5M dataset provides Geographic information associated with the collection sites of the organisms.
The following geographic data is presented in the country, province_state, latitude, and
longitude fields of the metadata file(s):

• Latitude and Longitude coordinates
• Country
• Province or State

Size Information

The BIOSCAN-5M dataset provides information about size of the organisms.
The size data is presented in the image_measurement_value, area_fraction, and
scale_factor fields of the metadata file(s):

• Image measurement value: Total number of pixels occupied by the organism

Furthermore, utilizing our cropping tool, we calculated the following information about size of the organisms:

• Area fraction: Fraction of the original image, the cropped image comprises.
• Scale factor: Ratio of the cropped image to the cropped and resized image.

Benchmark Experiments

Data Partitions

We partitioned the BIOSCAN-5M dataset into splits for both closed-world and open-world machine learning problems.
To use the partitions we propose, see the split field of the metadata file(s).

• The closed-world classification task uses samples labelled with a scientific name for their species
  (train, val, and test partitions).

  • This task requires the model to correctly classify new images and DNA barcodes of across a known set of species labels that were seen during training.

• The open-world classification task uses samples whose species name is a placeholder name,
  and whose genus name is a scientific name
  (key_unseen, val_unseen, and test_unseen partitions).

  • This task requires the model to correctly group together new species that were not seen during training.
  • In the retreival paradigm, this task can be performed using test_unseen records as queries against keys from the key_unseen records.
  • Alternatively, this data can be evaluated at the genus-level by classification via the species in the train partition.

• Samples labelled with placeholder species names, and whose genus name is not a scientific name are placed in the other_heldout partition.

  • This data can be used to train an unseen species novelty detector.

• Samples without species labels are placed in the pretrain partition, which comprises 90% of the data.

  • This data can be used for self-supervised or semi-supervised training.

Task-I: DNA-based taxonomic classification

Two stages of the proposed semi-supervised learning set-up based on BarcodeBERT.

1. Pretraining: DNA sequences are tokenized using non-overlapping k-mers and 50% of the tokens are masked for the MLM task.
   Tokens are encoded and fed into a transformer model. The output embeddings are used for token-level classification.
2. Fine-tuning: All DNA sequences in a dataset are tokenized using non-overlapping $k$-mer tokenization and all tokenized sequences, without masking, are passed through the pretrained transformer model. Global mean-pooling is applied over the token-level embeddings and the output is used for taxonomic classification.

Results

The performance of the taxonomic classification using DNA barcode sequences of the BIOSCAN-5M dataset is summarized as follows:

Performance of DNA-based sequence models in closed- and open-world settings.
For the closed-world setting, we show the species-level accuracy (%) for predicting seen species.
For the open-world setting, we show genus-level accuracy (%) for unseen species, while using seen species to fit the model.
Bold values indicate the best result, and italicized values indicate the second best.

Task-II: Zero-shot transfer learning

We follow the experimental setup recommended by zero-shot clustering,
expanded to operate on multiple modalities.

1. Take pretrained encoders.
2. Extract feature vectors from the stimuli by passing them through the pretrained encoder.
3. Reduce the embeddings with UMAP.
4. Cluster the reduced embeddings with Agglomerative Clustering.
5. Evaluate against the ground-truth annotations with Adjusted Mutual Information.

Results

The performance of the zero-shot transfer learning experiments on the BIOSCAN-5M dataset is summarized as follows:

Task-III: Multimodal retrieval learning

Our experiments using the CLIBD are conducted in two steps.

1. Training: Multiple modalities, including RGB images, textual taxonomy, and DNA sequences, are encoded separately,
   and trained using a contrastive loss function.
2. Inference: Image vs DNA embedding is used as a query, and compared to the embeddings obtained from a database of image,
   DNA and text (keys). The cosine similarity is used to find the closest key embedding, and the corresponding taxonomic label is used to classify the query.

Results

The performance of the multimodal retrieval learning experiments on the BIOSCAN-5M dataset is summarized as follows:

Copyright and License

The images and metadata included in the BIOSCAN-5M dataset available through this repository are subject to copyright
and licensing restrictions shown in the following:

• Copyright Holder: CBG Photography Group
• Copyright Institution: Centre for Biodiversity Genomics (email:[email protected])
• Photographer: CBG Robotic Imager
• Copyright License: Creative Commons Attribution 3.0 Unported (CC BY 3.0)
• Copyright Contact: [email protected]
• Copyright Year: 2021