2025-03-10 ワシントン大学セントルイス校
<関連情報>
- https://source.washu.edu/2025/03/multimodal-ai-tool-supports-study-of-ecosystems/
- https://engineering.washu.edu/news/2025/Multimodal-AI-tool-supports-ecological-applications.html
- https://arxiv.org/abs/2411.00683
- https://github.com/mvrl/TaxaBind
TaxaBind: 生態学アプリケーションのための統一された埋め込み空間 TaxaBind: A Unified Embedding Space for Ecological Applications
Srikumar Sastry, Subash Khanal, Aayush Dhakal, Adeel Ahmad, Nathan Jacobs
arxive Submitted on 1 Nov 2024
DOI:https://doi.org/10.48550/arXiv.2411.00683
Abstract
We present TaxaBind, a unified embedding space for characterizing any species of interest. TaxaBind is a multimodal embedding space across six modalities: ground-level images of species, geographic location, satellite image, text, audio, and environmental features, useful for solving ecological problems. To learn this joint embedding space, we leverage ground-level images of species as a binding modality. We propose multimodal patching, a technique for effectively distilling the knowledge from various modalities into the binding modality. We construct two large datasets for pretraining: iSatNat with species images and satellite images, and iSoundNat with species images and audio. Additionally, we introduce TaxaBench-8k, a diverse multimodal dataset with six paired modalities for evaluating deep learning models on ecological tasks. Experiments with TaxaBind demonstrate its strong zero-shot and emergent capabilities on a range of tasks including species classification, cross-model retrieval, and audio classification.
