9 November 2021 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Fine-grained image analysis (FGIA) is a longstanding and fundamental problem in computer vision and pattern recognition, and underpins a diverse set of real-world applications. The task of FGIA targets analyzing visual objects from subordinate categories, e.g., species of birds or models of cars. The small inter-class and large intra-class variation inherent to fine-grained image analysis makes it a challenging problem. Capitalizing on advances in deep learning, in recent years we have witnessed remarkable progress in deep learning powered FGIA. In this paper we present a systematic survey of these advances, where we attempt to re-define and broaden the field of FGIA by consolidating two fundamental fine-grained research areas – fine-grained image recognition and fine-grained image retrieval. In addition, we also review other key issues of FGIA, such as publicly available benchmark datasets and related domain-specific applications. We conclude by highlighting several research directions and open problems which need further exploration from the community.
8 March 2018 in PLOS Computational Biology
Passive acoustic sensing has emerged as a powerful tool for quantifying anthropogenic impacts on biodiversity, especially for echolocating bat species. To better assess bat population trends there is a critical need for accurate, reliable, and open source tools that allow the detection and classification of bat calls in large collections of audio recordings. The majority of existing tools are commercial or have focused on the species classification task, neglecting the important problem of first localizing echolocation calls in audio which is particularly problematic in noisy recordings. We developed a convolutional neural network based open-source pipeline for detecting ultrasonic, full-spectrum, search-phase calls produced by echolocating bats. Our deep learning algorithms were trained on full-spectrum ultrasonic audio collected along road-transects across Europe and labelled by citizen scientists from www.batdetective.org. When compared to other existing algorithms and commercial systems, we show significantly higher detection performance of search-phase echolocation calls with our test sets. As an example application, we ran our detection pipeline on bat monitoring data collected over five years from Jersey (UK), and compared results to a widely-used commercial system. Our detection pipeline can be used for the automatic detection and monitoring of bat populations, and further facilitates their use as indicator species on a large scale. Our proposed pipeline makes only a small number of bat specific design decisions, and with appropriate training data it could be applied to detecting other species in audio. A crucial novelty of our work is showing that with careful, non-trivial, design and implementation considerations, state-of-the-art deep learning methods can be used for accurate and efficient monitoring in audio. There is a critical need for robust and accurate tools to scale up biodiversity monitoring and to manage the impact of anthropogenic change. For example, the monitoring of bat species and their population dynamics can act as an important indicator of ecosystem health as they are particularly sensitive to habitat conversion and climate change. In this work we propose a fully automatic and efficient method for detecting bat echolocation calls in noisy audio recordings. We show that our approach is more accurate compared to existing algorithms and other commercial tools. Our method enables us to automatically estimate bat activity from multi-year, large-scale, audio monitoring programmes.
7 August 2024 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Multimodal learning, which aims to understand and analyze information from multiple modalities, has achieved substantial progress in the supervised regime in recent years. However, the heavy dependence on data paired with expensive human annotations impedes scaling up models. Meanwhile, given the availability of large-scale unannotated data in the wild, self-supervised learning has become an attractive strategy to alleviate the annotation bottleneck. Building on these two directions, self-supervised multimodal learning (SSML) provides ways to learn from raw multimodal data. In this survey, we provide a comprehensive review of the state-of-the-art in SSML, in which we elucidate three major challenges intrinsic to self-supervised learning with multimodal data: 1) learning representations from multimodal data without labels, 2) fusion of different modalities, and 3) learning with unaligned data. We then detail existing solutions to these challenges. Specifically, we consider 1) objectives for learning from multimodal unlabeled data via self-supervision, 2) model architectures from the perspective of different multimodal fusion strategies, and 3) pair-free learning strategies for coarse-grained and fine-grained alignment. We also review real-world applications of SSML algorithms in diverse fields, such as healthcare, remote sensing, and machine translation. Finally, we discuss challenges and future directions for SSML.
17 December 2024 in Trends in Ecology & Evolution
Artificial Intelligence (AI) is an emerging tool that could be leveraged to identify the effective conservation solutions demanded by the urgent biodiversity crisis. We present the results of our horizon scan of AI applications likely to significantly benefit biological conservation. An international panel of conservation scientists and AI experts identified 21 key ideas. These included species recognition to uncover 'dark diversity', multimodal models to improve biodiversity loss predictions, monitoring wildlife trade, and addressing human-wildlife conflict. We consider the potential negative impacts of AI adoption, such as AI colonialism and loss of essential conservation skills, and suggest how the conservation field might adapt to harness the benefits of AI while mitigating its risks.
1 April 2020 in ISPRS Journal of Photogrammetry and Remote Sensing
16 December 2022 in bioRxiv
SBSantiago Martínez Balvanera
Santiago Martínez Balvanera
+11
Acoustic monitoring is a scalable approach for assessing bat populations, yet automating the detection and classification of bat echolocation calls remains challenging, particularly in data-scarce regions. Although deep learning (DL) is increasingly applied to this task, most existing approaches repurpose computer-vision architectures and generate a single prediction per spectrogram clip, offering limited robustness to variable background noise and potentially constraining generality across regions and species assemblages. Here, we develop BatDetect2, an open-source DL pipeline for the joint detection and classification of bat echolocation calls. BatDetect2 builds on a 2D convolutional architecture and incorporates two targeted modifications: (i) a temporal self-attention layer designed to capture long-range structure across call sequences, and (ii) convolutional layers augmented with frequency coordinates to explicitly encode frequency information directly. We evaluate model generality using five diverse datasets from four different regions: UK, Mexico, Australia, and Brazil, and conduct ablation analyses using a UK dataset spanning 17 bat species. We further assess whether a trained model can detect echolocation calls from species absent from the training data. BatDetect2 consistently outperforms a traditional call-parameter extraction baseline across all datasets and evaluation metrics. Ablation analyses show that the inclusion of temporal self-attention yields a substantial species classification performance gain, increasing mean Average Precision (mAP) from 0.83 to 0.88, while frequency-coordinate augmentation provides no measurable benefit. When applied to novel species assemblages without retraining, model detection performance varies across datasets, with Average Precision ranging from 0.60 to 0.98. Overall, BatDetect2 demonstrates strong and transferable performance across acoustically and taxonomically diverse regions. By jointly detecting and classifying all bat calls present in each input clip, the pipeline provides a practical and extensible tool for passive acoustic monitoring. The full training pipeline and a pretrained UK model are released through the open-source Python package batdetect2 , enabling practitioners to develop and deploy models using their own data.