Accurate crop disease classification is crucial for ensuring food security and enhancing agricultural productivity. However, the existing crop disease classification algorithms primarily focus on a single image modality and typically require a large number of samples. Our research counters these issues by using pre-trained Vision–Language Models (VLMs), which enhance the multimodal synergy for better crop disease classification than the traditional unimodal approaches. Firstly, we apply the multimodal model Qwen-VL to generate meticulous textual descriptions for representative disease images selected through clustering from the training set, which will serve as prompt text for generating classifier weights. Compared to solely using the language model for prompt text generation, this approach better captures and conveys fine-grained and image-specific information, thereby enhancing the prompt quality. Secondly, we integrate cross-attention and SE (Squeeze-and-Excitation) Attention into the training-free mode VLCD(Vision-Language model for Crop Disease classification) and the training-required mode VLCD-T (VLCD-Training), respectively, for prompt text processing, enhancing the classifier weights by emphasizing the key text features. The experimental outcomes conclusively prove our method’s heightened classification effectiveness in few-shot crop disease scenarios, tackling the data limitations and intricate disease recognition issues. It offers a pragmatic tool for agricultural pathology and reinforces the smart farming surveillance infrastructure.

Accurate crop diseases classification is crucial for ensuring food security and enhancing agricultural productivity. However, existing crop disease classification algorithms primarily focus on a single image modality and typically require a large number of samples. Our research counters these issues by using pre-trained Vision-Language Models (VLMs), which enhances multimodal synergy for better crop disease classification than traditional unimodal approaches. Firstly, we apply the multimodal model Qwen-VL to generate meticulous textual descriptions for representative disease images selected through clustering from the training set, which will serve as prompt text for generating classifier weights. Compared to using solely the language model for prompt text generation, this approach better captures and conveys fine-grained and image-specific information, thereby enhancing prompt quality. Secondly, we integrate the cross-attention and the SE (Squeeze-and-Excitation) attention into the training-free mode VLCD (Vision-Language model for Crop Diseases classification) and the training-required mode VLCD-T (VLCD-Training) respectively for prompt text processing, enhancing classifier weights by emphasizing key text features. Experimental outcomes conclusively prove our method’s heightened classification effectiveness in few-shot crop disease scenarios, tackling data limitations and intricate disease recognition issues. It offers a pragmatic tool for agricultural pathology and reinforces the smart farming surveillance infrastructure.

Ensuring precise segmentation of colorectal polyps holds critical importance in the early diagnosis and treatment of colorectal cancer. Nevertheless, existing deep learning-based segmentation methods are fully supervised, requiring extensive, precise, manual pixel-level annotation data, which leads to high annotation costs. Additionally, it remains challenging to train large-scale segmentation models when confronted with limited colonoscopy data. To address these issues, we introduce the general segmentation foundation model—the Segment Anything Model (SAM)—into the field of medical image segmentation. Fine-tuning the foundation model is an effective approach to tackle sample scarcity. However, current SAM fine-tuning techniques still rely on precise annotations. To overcome this limitation, we propose WSPolyp-SAM, a novel weakly supervised approach for colonoscopy polyp segmentation. WSPolyp-SAM utilizes weak annotations to guide SAM in generating segmentation masks, which are then treated as pseudo-labels to guide the fine-tuning of SAM, thereby reducing the dependence on precise annotation data. To improve the reliability and accuracy of pseudo-labels, we have designed a series of enhancement strategies to improve the quality of pseudo-labels and mitigate the negative impact of low-quality pseudo-labels. Experimental results on five medical image datasets demonstrate that WSPolyp-SAM outperforms current fully supervised mainstream polyp segmentation networks on the Kvasir-SEG, ColonDB, CVC-300, and ETIS datasets. Furthermore, by using different amounts of training data in weakly supervised and fully supervised experiments, it is found that weakly supervised fine-tuning can save 70% to 73% of annotation time costs compared to fully supervised fine-tuning. This study provides a new perspective on the combination of weakly supervised learning and SAM models, significantly reducing annotation time and offering insights for further development in the field of colonoscopy polyp segmentation.

In recent years, the rapid growth of remote sensing (RS) datasets has greatly enhanced the difficulty of RS image classification tasks. Given the remarkable success of large visual-language models (VLMs) in various domains, their potential for RS image classification tasks has attracted significant attention. In this study, we focus on Contrastive Language-Image Pre-training (CLIP), a widely recognized contrastive learning model known for its strong zero-shot transferability. However, the domain gap between the pre-training data (e.g., natural images) of CLIP and RS data can significantly hinder the application of CLIP. First, the scarcity of labeled RS data poses a significant challenge for fine-tuning CLIP. Second, the complexity and diversity of RS images substantially increase the fine-tuning time and computational resources needed to individually adapt CLIP to new tasks. Third, independently fine-tuning CLIP for each RS dataset ignores the inter-task relationships that frequently exist among remote sensing images from various sources, leading to reduced classification accuracy. To address these challenges, we propose a learnable Mixture of Adapters (MoA) module in the visual branch of CLIP, resulting in the CLIP-MoA framework. Inspired by the Mixture of Experts (MoE), the MoA module models inter-dataset relationships by integrating multiple adapters through a gating network, while preserving original information via a residual connection between the MoA input and the fine-tuned features. This module enables CLIP-MoA to effectively capture both inter-task commonalities and task-specific characteristics by fine-tuning a minimal set of MoA parameters across multiple datasets simultaneously, thereby reducing computational overhead and fine-tuning time while improving classification accuracy. Comprehensive experiments conducted on a multi-task benchmark involving 11 remote sensing image classification datasets demonstrate that CLIP-MoA achieves significant improvements in both classification accuracy and generalization capability compared to zero-shot CLIP, linear probe CLIP, CLIP-Adapter, and CoOp.

Automatic and accurate tooth segmentation on 3D dental point clouds plays a pivotal role in computer-aided dentistry. Existing Transformer-based methods focus on aggregating local features, but fail to directly model global contexts due to memory limitations and high computational cost. In this paper, we propose a novel Transformer-based 3D tooth segmentation network, called PointRegion, which can process the entire point cloud at a low cost. Following a novel modeling of semantic segmentation that interprets the point cloud as a tessellation of learnable regions, we first design a RegionPartition module to partition the 3D point cloud into a certain number of regions and project these regions as embeddings in an effective way. Then, an offset-attention based RegionEncoder module is applied on all region embeddings to model global context among regions and predict the class logits for each region. Considering the irregularity and disorder of 3D point cloud data, a novel mechanism is proposed to build the point-to-region association to replace traditional convolutional operations. The mechanism, as a medium between points and regions, automatically learns the probabilities that each point belongs to its neighboring regions from the similarity between point and region features, achieving the goal of point-level segmentation. Since the number of regions is far less than the number of points, our proposed PointRegion model can leverage the capability of the global-based Transformer on large-scale point clouds with low computational cost and memory consumption. Finally, extensive experiments demonstrate the effectiveness and superiority of our method on our dental dataset.

Fire is a disastrous event that causes immense dangers. Conventional fire detection methods use sensors in indoor settings. By contrast, computer vision-based methods have made substantial advancements in terms of detection accuracy, speed, and versatility. However, existing approaches often treat fire detection as image classification, image segmentation, and object detection problems, overlooking the characteristic of smoke, including its floatability, ambiguity, and transparency. To foster advancements in this field, we formalize fire detection as the image matting problem and introduce a novel smoke and fire matting dataset SFMatting-800. This dataset not only features high-quality annotations on transparency but also encompasses diverse information regarding scene types and fire attributes. These annotations serve as valuable resources for training, evaluating image matting models, and expanding data through compositing with a substantial background dataset. With SFMatting-800, we have conducted comprehensive evaluations using traditional image matting methods, trimap-based and trimap-free deep image matting methods. The experimental results not only demonstrate the value of our dataset for the fine-grained fire detection task, but also help to expose the problem of existing matting methods in processing extensive regions with high transparency and misidentification between background and targets. The dataset will be made publicly available.