{"id":316952,"name":"CLIBD","description":"A model uses contrastive learning to map biological images, DNA barcodes, and textual taxonomic labels to the same latent space.","url":"https://github.com/bioscan-ml/clibd","last_synced_at":"2026-05-31T15:30:22.651Z","repository":{"id":241447075,"uuid":"806314505","full_name":"bioscan-ml/clibd","owner":"bioscan-ml","description":"A multimodal model bridging vision and genomics for biodiversity monitoring at scale.","archived":false,"fork":false,"pushed_at":"2026-05-11T19:17:35.000Z","size":15404,"stargazers_count":18,"open_issues_count":2,"forks_count":4,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-05-14T07:06:29.748Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bioscan-ml.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-05-27T00:16:21.000Z","updated_at":"2026-04-05T23:59:06.000Z","dependencies_parsed_at":"2024-05-28T12:01:01.836Z","dependency_job_id":"b9317f59-ce2e-47ba-9207-760ed937dda3","html_url":"https://github.com/bioscan-ml/clibd","commit_stats":null,"previous_names":["3dlg-hcvc/bioscan-clip","bioscan-ml/clibd"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/bioscan-ml/clibd","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bioscan-ml%2Fclibd","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bioscan-ml%2Fclibd/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bioscan-ml%2Fclibd/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bioscan-ml%2Fclibd/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bioscan-ml","download_url":"https://codeload.github.com/bioscan-ml/clibd/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bioscan-ml%2Fclibd/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33432866,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-23T22:14:44.296Z","status":"online","status_checked_at":"2026-05-24T02:00:06.296Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"owner":{"login":"bioscan-ml","name":"BIOSCAN","uuid":"175227258","kind":"organization","description":"Illuminating biodiversity with DNA-based identification systems","email":"contact@bioscancanada.org","website":"https://biodiversitygenomics.net/research/bioscan/","location":null,"twitter":null,"company":null,"icon_url":"https://avatars.githubusercontent.com/u/175227258?v=4","repositories_count":1,"last_synced_at":"2024-10-29T05:17:09.632Z","metadata":{"has_sponsors_listing":false},"html_url":"https://github.com/bioscan-ml","funding_links":[],"total_stars":10,"followers":3,"following":0,"created_at":"2024-10-29T05:17:09.653Z","updated_at":"2024-10-29T05:17:09.653Z","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bioscan-ml","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bioscan-ml/repositories"},"packages":[],"commits":{"id":10034432,"full_name":"bioscan-ml/clibd","default_branch":"main","total_commits":612,"total_committers":8,"total_bot_commits":0,"total_bot_committers":0,"mean_commits":76.5,"dds":0.18300653594771243,"past_year_total_commits":14,"past_year_total_committers":2,"past_year_total_bot_commits":0,"past_year_total_bot_committers":0,"past_year_mean_commits":7.0,"past_year_dds":0.0714285714285714,"last_synced_at":"2026-05-28T14:02:06.236Z","last_synced_commit":"363aae0ca145de6be72d9c4eaa1253ca14a63548","created_at":"2025-05-22T00:10:54.149Z","updated_at":"2026-05-28T14:01:53.734Z","committers":[{"name":"zmgong","email":"ming2280089874@gmail.com","login":"zmgong","count":500},{"name":"rust-in","email":"huoxiaoliangchn@163.com","login":"xl-huo","count":42},{"name":"angelxuanchang","email":"angelx@gmail.com","login":"angelxuanchang","count":20},{"name":"Chuanqi","email":"1345308560@qq.com","login":"charlie1587","count":17},{"name":"Scott Lowe","email":"scott.code.lowe@gmail.com","login":"scottclowe","count":15},{"name":"mga113","email":"mga113@cs-3dlg-01.cmpt.sfu.ca","login":null,"count":13},{"name":"Austin Wang","email":"austin.t.wang@gmail.com","login":"atwang16","count":4},{"name":"cta156","email":"cta156@cs-venus-08.cmpt.sfu.ca","login":null,"count":1}],"past_year_committers":[{"name":"zmgong","email":"ming2280089874@gmail.com","login":"zmgong","count":13},{"name":"Chuanqi","email":"1345308560@qq.com","login":"charlie1587","count":1}],"commits_url":"https://commits.ecosyste.ms/api/v1/hosts/GitHub/repositories/bioscan-ml%2Fclibd/commits","host":{"name":"GitHub","url":"https://github.com","kind":"github","last_synced_at":"2026-05-30T00:00:22.241Z","repositories_count":6248886,"commits_count":883727689,"contributors_count":34993189,"owners_count":1160619,"icon_url":"https://github.com/github.png","host_url":"https://commits.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://commits.ecosyste.ms/api/v1/hosts/GitHub/repositories"}},"issues_stats":{"full_name":"bioscan-ml/clibd","html_url":"https://github.com/bioscan-ml/clibd","last_synced_at":"2026-05-20T10:02:49.787Z","status":"error","issues_count":11,"pull_requests_count":38,"avg_time_to_close_issue":314121.3333333333,"avg_time_to_close_pull_request":278879.65625,"issues_closed_count":3,"pull_requests_closed_count":32,"pull_request_authors_count":4,"issue_authors_count":3,"avg_comments_per_issue":0.18181818181818182,"avg_comments_per_pull_request":0.15789473684210525,"merged_pull_requests_count":32,"bot_issues_count":0,"bot_pull_requests_count":0,"past_year_issues_count":2,"past_year_pull_requests_count":3,"past_year_avg_time_to_close_issue":16831.0,"past_year_avg_time_to_close_pull_request":58896.0,"past_year_issues_closed_count":1,"past_year_pull_requests_closed_count":2,"past_year_pull_request_authors_count":2,"past_year_issue_authors_count":2,"past_year_avg_comments_per_issue":0.5,"past_year_avg_comments_per_pull_request":0.0,"past_year_bot_issues_count":0,"past_year_bot_pull_requests_count":0,"past_year_merged_pull_requests_count":2,"created_at":"2025-05-22T00:10:58.095Z","updated_at":"2026-05-20T10:02:49.788Z","repository_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/repositories/bioscan-ml%2Fclibd","issues_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/repositories/bioscan-ml%2Fclibd/issues","issue_labels_count":{"bug":1},"pull_request_labels_count":{},"issue_author_associations_count":{"COLLABORATOR":9,"MEMBER":1,"NONE":1},"pull_request_author_associations_count":{"COLLABORATOR":31,"MEMBER":5,"NONE":2},"issue_authors":{"zmgong":9,"Thundermean-sky":1,"annavik":1},"pull_request_authors":{"zmgong":26,"charlie1587":6,"scottclowe":5,"Copilot":1},"host":{"name":"GitHub","url":"https://github.com","kind":"github","last_synced_at":"2026-05-22T00:00:21.018Z","repositories_count":14662835,"issues_count":34135897,"pull_requests_count":111729993,"authors_count":11269356,"icon_url":"https://github.com/github.png","host_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/repositories","owners_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/owners","authors_url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/authors"},"past_year_issue_labels_count":{},"past_year_pull_request_labels_count":{},"past_year_issue_author_associations_count":{"MEMBER":1,"NONE":1},"past_year_pull_request_author_associations_count":{"COLLABORATOR":2,"NONE":1},"past_year_issue_authors":{"annavik":1,"Thundermean-sky":1},"past_year_pull_request_authors":{"charlie1587":2,"Copilot":1},"maintainers":[{"login":"zmgong","count":35,"url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/authors/zmgong"},{"login":"charlie1587","count":5,"url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/authors/charlie1587"},{"login":"scottclowe","count":5,"url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/authors/scottclowe"},{"login":"annavik","count":1,"url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/authors/annavik"}],"active_maintainers":[{"login":"charlie1587","count":2,"url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/authors/charlie1587"},{"login":"annavik","count":1,"url":"https://issues.ecosyste.ms/api/v1/hosts/GitHub/authors/annavik"}]},"events":{"total":{"ReleaseEvent":1,"DeleteEvent":10,"MemberEvent":1,"PullRequestEvent":29,"ForkEvent":3,"IssuesEvent":20,"WatchEvent":5,"IssueCommentEvent":7,"PushEvent":321,"PullRequestReviewCommentEvent":3,"PullRequestReviewEvent":6,"CreateEvent":16},"last_year":{"DeleteEvent":1,"PullRequestEvent":1,"IssuesEvent":1,"WatchEvent":1,"PushEvent":5,"PullRequestReviewEvent":1,"PullRequestReviewCommentEvent":1,"CreateEvent":1}},"keywords":[],"dependencies":[{"ecosystem":"pypi","filepath":"requirements.txt","sha":null,"kind":"manifest","created_at":"2024-05-28T12:01:00.801Z","updated_at":"2024-05-28T12:01:00.801Z","repository_link":"https://github.com/bioscan-ml/clibd/blob/main/requirements.txt","dependencies":[{"id":18262568047,"package_name":"numpy","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false},{"id":18262568048,"package_name":"matplotlib","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false},{"id":18262568049,"package_name":"pandas","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false},{"id":18262568050,"package_name":"scikit-learn","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false},{"id":18262568051,"package_name":"tqdm","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false},{"id":18262568052,"package_name":"transformers","ecosystem":"pypi","requirements":"==4.29.2","direct":true,"kind":"runtime","optional":false},{"id":18262568053,"package_name":"wandb","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false},{"id":18262568054,"package_name":"h5py","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false},{"id":18262568055,"package_name":"chardet","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false},{"id":18262568056,"package_name":"ftfy","ecosystem":"pypi","requirements":"==6.1.1","direct":true,"kind":"runtime","optional":false},{"id":18262568057,"package_name":"timm","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false},{"id":18262568058,"package_name":"hydra-core","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false},{"id":18262568059,"package_name":"seaborn","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false},{"id":18262568060,"package_name":"pillow","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false},{"id":18262568061,"package_name":"scipy","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false},{"id":18262568062,"package_name":"safetensors","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false},{"id":18262568063,"package_name":"omegaconf","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false},{"id":18262568064,"package_name":"edgegpt","ecosystem":"pypi","requirements":"*","direct":true,"kind":"runtime","optional":false},{"id":18262568173,"package_name":"plotly","ecosystem":"pypi","requirements":"==5.18.0","direct":true,"kind":"runtime","optional":false},{"id":18262568174,"package_name":"umap-learn","ecosystem":"pypi","requirements":"==0.5.5","direct":true,"kind":"runtime","optional":false}]},{"ecosystem":"pypi","filepath":"setup.py","sha":null,"kind":"manifest","created_at":"2024-05-28T12:01:01.510Z","updated_at":"2024-05-28T12:01:01.510Z","repository_link":"https://github.com/bioscan-ml/clibd/blob/main/setup.py","dependencies":[]}],"score":5.075173815233827,"created_at":"2025-05-22T00:10:50.732Z","updated_at":"2026-05-31T15:30:22.653Z","avatar_url":"https://github.com/bioscan-ml.png","language":"Python","category":"Biosphere","sub_category":"Biodiversity Analysis and Metrics","monthly_downloads":0,"total_dependent_repos":0,"total_dependent_packages":0,"readme":"# CLIBD: Bridging Vision and Genomics for Biodiversity Monitoring at Scale\nThis is the official implementation for \"CLIBD: Bridging Vision and Genomics for Biodiversity Monitoring at Scale\".\nLinks: [website](https://bioscan-ml.github.io/clibd/) | [paper](https://arxiv.org/abs/2405.17537)\n\n# Overview\n![Teaser](./docs/static/images/method.svg)\nTaxonomically classifying organisms at scale is crucial for monitoring biodiversity, understanding ecosystems, and preserving sustainability.  It is possible to taxonomically classify organisms based on their image or their [DNA barcode](https://en.wikipedia.org/wiki/DNA_barcoding).  While DNA barcodes are precise at species identification, they are less readily available than images.  Thus, we investigate whether we can use DNA barcodes to improve taxonomic classification using image.  \n\nWe introduce CLIBD, a model uses contrastive learning to map biological images, DNA barcodes, and textual taxonomic labels to the same latent space.  The model is initialized using pretrained encoders for images ([vit-base-patch16-224](https://huggingface.co/google/vit-base-patch16-224)), DNA barcodes ([BarcodeBERT](https://github.com/Kari-Genomics-Lab/BarcodeBERT)), and textual taxonomic labels ([BERT-small](https://huggingface.co/prajjwal1/bert-small)), and the weights of the encoders are fine-tuned using LoRA.\nThe aligned image-DNA embedding space improves taxonomic classification using images and allows us to do cross-modal retrieval from image to DNA. We train CLIBD on the [BIOSCAN-1M](https://biodiversitygenomics.net/projects/1m-insects/) and [BIOSCAN-5M](https://biodiversitygenomics.net/projects/5m-insects/) insect datasets.  These datasets provides paired images of insects and their DNA barcodes, along with their taxonomic labels.  \n\n# Setup environment\nCLIBD was developed using Python 3.10 and PyTorch 2.0.1.  We recommend the use of GPU and CUDA for efficient training and inference.  Our models were developed with CUDA 11.7 and 12.4.  \nWe also recommend the use of [miniconda](https://docs.anaconda.com/miniconda/miniconda-install/) for managing your environments. \n\nTo setup the environment width necessary dependencies, type the following commands:\n```shell\nconda create -n CLIBD python=3.10 -y\nconda activate CLIBD\nconda install pytorch=2.0.1 torchvision=0.15.2 torchtext=0.15.2 pytorch-cuda=11.7 -c pytorch -c nvidia -y\npip install -r requirements.txt\npip install -e .\npip install git+https://github.com/Baijiong-Lin/LoRA-Torch\n```\n\n\n\nDepending on your GPU version, you may have to modify the torch version and other package versions in [requirements.txt](./requirements.txt).\n\n# Pretrained embeddings and models\nWe provide pretrained embeddings and model weights.  We evaluate our models by encoding the image or DNA barcode, and using the taxonomic labels from the closest matching embedding (either using image or DNA barcode).  See [Download dataset](#download-dataset) and [Running Experiments](#running-experiments) for how to get the data, and to train and evaluate the models.\n\n| Training data |  Aligned modalities |  Embeddings |  Model  | Config |\n|---------------|---------------------|-------------|---------|--------|\n| BIOSCAN-1M    |  None               |  [Embedding](https://aspis.cmpt.sfu.ca/projects/bioscan/checkpoint/for_readme/ver_1_0/no_alignment/extracted_feature_from_test_split.hdf5) |  N/A   |  [Link](https://github.com/bioscan-ml/clibd/blob/45f38fc/bioscanclip/config/model_config/for_bioscan_1m/final_experiments/image_dna_text_no_loading.yaml)  |\n| BIOSCAN-1M    |  **I**mage + **D**NA        |  [Embedding](https://aspis.cmpt.sfu.ca/projects/bioscan/checkpoint/for_readme/ver_1_0/image_dna/extracted_feature_from_test_split.hdf5)|  [Link](https://aspis.cmpt.sfu.ca/projects/bioscan/checkpoint/for_readme/ver_1_0/image_dna/best.pth)   |  [Link](https://github.com/bioscan-ml/clibd/blob/45f38fc/bioscanclip/config/model_config/for_bioscan_1m/final_experiments/image_dna_seed_42.yaml)  |\n| BIOSCAN-1M    |  **I**mage + **D**NA + **T**ax  |  [Embedding](https://aspis.cmpt.sfu.ca/projects/bioscan/checkpoint/for_readme/ver_1_0/image_dna_text/extracted_feature_from_test_split.hdf5)       |  [Link](https://aspis.cmpt.sfu.ca/projects/bioscan/checkpoint/for_readme/ver_1_0/image_dna_text/best.pth) |  [Link](https://github.com/bioscan-ml/clibd/blob/45f38fc/bioscanclip/config/model_config/for_bioscan_1m/final_experiments/image_dna_text_seed_42.yaml) |\n| BIOSCAN-5M    |  None               |  [Embedding](https://aspis.cmpt.sfu.ca/projects/bioscan/checkpoint/for_readme/ver_1_0/bioscan_5m/no_alignment/extracted_feature_from_test_split.hdf5)       |  N/A   |  [Link](https://github.com/bioscan-ml/clibd/blob/45f38fc/bioscanclip/config/model_config/for_bioscan_5m/no_alignment_baseline/no_align.yaml)  |\n| BIOSCAN-5M    |  **I**mage + **D**NA        |  [Embedding](https://aspis.cmpt.sfu.ca/projects/bioscan/checkpoint/for_readme/ver_1_0/bioscan_5m/image_dna/extracted_feature_from_test_split.hdf5)      |  [Link](https://aspis.cmpt.sfu.ca/projects/bioscan/checkpoint/for_readme/ver_1_0/bioscan_5m/image_dna/best.pth)    |  [Link](https://github.com/bioscan-ml/clibd/blob/45f38fc/bioscanclip/config/model_config/for_bioscan_5m/final_experiments/image_dna_seed_42.yaml)  |\n| BIOSCAN-5M    |  **I**mage + **D**NA + **T**ax  |  [Embedding](https://aspis.cmpt.sfu.ca/projects/bioscan/checkpoint/for_readme/ver_1_0/bioscan_5m/image_dna_text/extracted_feature_from_test_split.hdf5)      |  [Link](https://aspis.cmpt.sfu.ca/projects/bioscan/checkpoint/for_readme/ver_1_0/bioscan_5m/image_dna_text/best.pth)|   [Link](https://github.com/bioscan-ml/clibd/blob/45f38fc/bioscanclip/config/model_config/for_bioscan_5m/final_experiments/image_dna_text_seed_42.yaml)  |\n\nWe also provide checkpoints trained with LoRA layers. You can download them from this [Link](https://aspis.cmpt.sfu.ca/projects/bioscan/clip_project/ckpt/bioscan_clip/lora_version_ckpt.zip)\n\n## Quick start\nInstead of conducting a full training, you can choose to download pre-trained models or pre-extracted embeddings for evaluation from the table. You may need to posistion the downloaded checkpoints and extracted features in to the proper position based on the config file.\n\n# Download dataset\n![Data Partioning Visual](./docs/static/images/partition.svg) \u003cbr\u003e\nFor BIOSCAN 1M, we partition the dataset for our CLIBD experiments into a training set for contrastive learning, and validation and test partitions. The training set has records without any species labels as well as a set of seen species. The validation and test sets include seen and unseen species. These images are further split into subpartitions of queries and keys for evaluation.\n\nFor BIOSCAN 5M, we use the dataset partitioning established in the BIOSCAN-5M paper.\n\nFor training and reproducing our experiments, we provide HDF5 files with BIOSCAN-1M and BIOSCAN-5M images.  See [DATA.md](DATA.md) for format details. We also provide scripts for generating the HDF5 files directly from the BIOSCAN-1M and BIOSCAN-5M data.\n\n### Download BIOSCAN-1M data (79.7 GB)\n```shell\n# From project folder\nmkdir -p data/BIOSCAN_1M/split_data\ncd data/BIOSCAN_1M/split_data\nwget https://aspis.cmpt.sfu.ca/projects/bioscan/clip_project/data/version_0.2.1/BioScan_data_in_splits.hdf5\n```\n\n### Download BIOSCAN-5M data (190.4 GB)\n```shell\n# From project folder\nmkdir -p data/BIOSCAN_5M\ncd data/BIOSCAN_5M\nwget https://aspis.cmpt.sfu.ca/projects/bioscan/BIOSCAN_CLIP_for_downloading/BIOSCAN_5M.hdf5\n```\nFor more information about the hdf5 files, please check [DATA.md](DATA.md).\n\nYou can also download the processed data by checking our [huggimgface repo](https://huggingface.co/datasets/bioscan-ml/bioscan-clibd)\n\n### Download data for generating hdf5 files\n\nYou can check [BIOSCAN-1M](https://github.com/zahrag/BIOSCAN-1M) and [BIOSCAN-5M](https://github.com/zahrag/BIOSCAN-5M) to download tsv files. But they are actually not necessary.\n\n# Running experiments\nWe recommend the use of [weights and biases](https://wandb.ai/site) to track and log experiments\n\n## Activate Wandb\n#### Register/Login for a [free wandb account](https://wandb.ai/site)\n```shell\nwandb login\n# Paste your wandb's API key\n```\nNote: To enable wandb, you also need to modify /bioscanclip/config/global_config and set:\n```yaml\ndebug_flag: false\n```\n\n\n## Checkpoints\n\nDownload checkpoint for BarcodeBERT and bioscan_clip and place them under `ckpt`.  \n```shell\npip install huggingface-cli\n# From project folder\nhuggingface-cli download bioscan-ml/clibd --include \"ckpt/*\" --local-dir .\n```\n\nYou can also check this [link](https://huggingface.co/bioscan-ml/clibd/tree/main) to download the files manually.\n\n## Train\n\nUse [train_cl.py](./scripts/train_cl.py) with the appropriate `model_config` to train CLIBD.\n```shell\n# From project folder\npython scripts/train_cl.py 'model_config={config_name}'\n```\n\nTo train the full model (I+D+T) using BIOSCAN-1M:\n```shell\n# From project folder\npython scripts/train_cl.py 'model_config=for_bioscan_1m/final_experiments/image_dna_text_seed_42.yaml'\n```\nFor multi-GPU training, you may need to specify the transport communication between the GPU using NCCL_P2P_LEVEL:\n```shell\nNCCL_P2P_LEVEL=NVL python scripts/train_cl.py 'model_config=for_bioscan_1m/final_experiments/image_dna_text_seed_42.yaml'\n```\n\nFor example, using the following command, you can load the pre-trained ViT-B, BarcodeBERT, and BERT-small and fine-tune them through contrastive learning. Note that this training will only update their LoRA layers, not all the parameters.\n```shell\npython scripts/train_cl.py 'model_config=for_bioscan_5m/lora_vit_lora_barcode_bert_lora_bert_5m_no_loading.yaml'\n```\n\n## Evaluation\n\nDuring evaluation, we using the trained encoders to obtain embeddings for input image or DNA, and the find the closest matching image or DNA and use the corresponding taxonomical labels as the predicted labels.  We report both the micro and class averaged accuracy for seen and unseen species. \n\nTo run evaluation for BIOSCAN-1M:\n```shell\n# From project folder\npython scripts/inference_and_eval.py 'model_config=for_bioscan_1m/final_experiments/image_dna_text_seed_42.yaml'\n```\n\nTo run evaluation for BIOSCAN-5M:\n```shell\npython scripts/inference_and_eval.py 'model_config=for_bioscan_5m/final_experiments/image_dna_text_seed_42.yaml'\n```\n\n## For BZSL experiment with the INSECT dataset.\n\nTo download unprocessed INSECT dataset, you can reference [BZSL](https://github.com/sbadirli/Fine-Grained-ZSL-with-DNA): \n\n```shell\nmkdir -p data/INSECT\ncd data/INSECT\n# Download the images and metadata here.\n\n\n# Note that we need to get the other three labels because the INSECT dataset only has the species label.\n# For that, please edit get_all_species_taxo_labels_dict_and_save_to_json.py, change Entrez.email = None to your email \npip install biopython\npython get_all_species_taxo_labels_dict_and_save_to_json.py\n\n# Then, generate CSV and hdf5 file for the dataset.\npython process_insect_dataset.py\n```\nThe downloaded data should be organized in this way:\n```shell\ndata\n├── INSECT\n│   ├── att_splits.mat\n│   ├── res101.mat\n│   ├── images\n│   │   │   ├── Abax parallelepipedus\n│   │   │   │   ├── BC_ZSM_COL_02878+1311934584.jpg\n│   │   │   │   ├── BC_ZSM_COL_05487+1338577126.JPG\n│   │   │   │   ├── ...\n│   │   │   ├── Abax parallelus\n│   │   │   ├── Acordulecera dorsalis\n│   │   │   ├── ...\n```\n\nYou can also download the processed file with:\n```shell\nwget https://aspis.cmpt.sfu.ca/projects/bioscan/BIOSCAN_CLIP_for_downloading/INSECT_data/processed_data.zip\nunzip processed_data.zip\n```\n\n### Train CLIBD with INSECT dataset\n```shell\npython scripts/train_cl.py 'model_config=for_bioscan_1m/lora_vit_lora_barcode_bert_lora_bert_ssl_on_insect.yaml'\n```\n\n###  Extract image and DNA features of INSECT dataset.\n\nTo perform contrastive learning for fine-tuning on the INSECT dataset.\n\n```shell\npython scripts/train_cl.py 'model_config=for_bioscan_1m/fine_tune_on_INSECT_dataset/image_dna_text_seed_42_on_INSECT_dataset.yaml'\n```\n\nTo perform supervise fine-tune image encoder with INSECT dataset.\n```shell\npython scripts/BZSL/fine_tune_bioscan_clip_image_on_insect.py 'model_config=for_bioscan_1m/final_experiments/image_dna_text_seed_42.yaml'\n```\n\nFor feature extracting\n\n```shell\npython scripts/extract_feature_for_insect_dataset.py 'model_config=for_bioscan_1m/fine_tune_on_INSECT_dataset/image_dna_text_seed_42_on_INSECT_dataset.yaml'\n```\nThen, you may move the extracted features to the BZSL folder or download the pre-extracted feature.\n\n```shell\nmkdir -p Fine-Grained-ZSL-with-DNA/data/INSECT/embeddings_from_bioscan_clip\ncp extracted_embedding/INSECT/dna_embedding_from_bioscan_clip.csv Fine-Grained-ZSL-with-DNA/data/INSECT/embeddings_from_bioscan_clip/dna_embedding_from_bioscan_clip.csv\ncp extracted_embedding/INSECT/image_embedding_from_bioscan_clip.csvFine-Grained-ZSL-with-DNA/data/INSECT/embeddings_from_bioscan_clip/image_embedding_from_bioscan_clip.csv\n```\n\n###  Run BZSL for evaluation.\n```shell\ncd Fine-Grained-ZSL-with-DNA/BZSL-Python\npython Demo.py --using_bioscan_clip_image_feature --datapath ../data --side_info dna_bioscan_clip --alignment --tuning\n```\n\n###  Flatten the `results.csv`.\n```shell\npython scripts/flattenCsv.pya -i PATH_TO_RESULTS_CSV -o PATH_TO_FLATTEN_CSV\n```\n\n# Citing CLIBD\nIf you use CLIBD in your research, please cite:\n```bibtex\n@inproceedings{gong2025clibd,\n    title={{CLIBD}: Bridging Vision and Genomics for Biodiversity Monitoring at Scale},\n    author={ZeMing Gong and Austin Wang and Xiaoliang Huo and Joakim Bruslund Haurum\n        and Scott C. Lowe and Graham W. Taylor and Angel X Chang\n    },\n    booktitle={The Thirteenth International Conference on Learning Representations},\n    year={2025},\n    url={https://openreview.net/forum?id=d5HUnyByAI},\n}\n```\n\n# Version log\n\n## Version 1.0 (Current version)\n\n- Release for the initial ICLR camera-ready submission\n- Support loading the checkpoint from Hugging Face when no checkpoint is found in the local path\n\n\n# Acknowledgements\nWe would like to express our gratitude for the use of the INSECT dataset, which played a pivotal role in the completion of our experiments. Additionally, we acknowledge the use and modification of code from the [Fine-Grained-ZSL-with-DNA](https://github.com/sbadirli/Fine-Grained-ZSL-with-DNA) repository, which facilitated part of our experimental work. The contributions of these resources have been invaluable to our project, and we appreciate the efforts of all developers and researchers involved.\n\nThis reseach was supported by the Government of Canada’s New Frontiers in Research Fund (NFRF) [NFRFT-2020-00073], \nCanada CIFAR AI Chair grants, and the Pioneer Centre for AI (DNRF grant number P1).\nThis research was also enabled in part by support provided by the Digital Research Alliance of Canada (alliancecan.ca).\n","funding_links":[],"readme_doi_urls":[],"works":{},"citation_counts":{},"total_citations":0,"keywords_from_contributors":[],"project_url":"https://ost.ecosyste.ms/api/v1/projects/316952","html_url":"https://ost.ecosyste.ms/projects/316952"}