Multimodal emotion recognition has emerged as a promising approach to capture the complex nature of human emotions by integrating information from various sources such as physiological signals, visual behavioral cues, and audio-visual content. However, current methods often struggle with effectively processing redundant or conflicting information across modalities and may overlook implicit inter-modal correlations. To address these challenges, this paper presents a novel multimodal emotion recognition framework which integrates audio-visual features with viewers’ EEG data to enhance emotion classification accuracy. The proposed approach employs modality-specific encoders to extract spatiotemporal features, which are then aligned through contrastive learning to capture inter-modal relationships. Additionally, cross-modal attention mechanisms are incorporated for effective feature fusion across modalities. The framework, comprising pre-training, fine-tuning, and testing phases, is evaluated on multiple datasets of emotional responses. The experimental results demonstrate that the proposed multimodal approach, which combines audio-visual features with EEG data, is highly effective in recognizing emotions, highlighting its potential for advancing emotion recognition systems.

In this paper, we addressed the challenges in sorting high-yield apple cultivars that traditionally relied on manual labor or system-based defect detection. Existing single-camera methods failed to uniformly capture the entire surface of apples, potentially leading to misclassification due to defects in unscanned areas. Various methods were proposed where apples were rotated using rollers on a conveyor. However, since the rotation was highly random, it was difficult to scan the apples uniformly for accurate classification. To overcome these limitations, we proposed a multi-camera-based apple sorting system with a rotation mechanism that ensured uniform and accurate surface imaging. The proposed system applied a rotation mechanism to individual apples while simultaneously utilizing three cameras to capture the entire surface of the apples. This method offered the advantage of quickly and uniformly acquiring the entire surface compared to single-camera and random rotation conveyor setups. The images captured by the system were analyzed using a CNN classifier deployed on embedded hardware. To maintain excellent CNN classifier performance while reducing its size and inference time, we employed knowledge distillation techniques. The CNN classifier demonstrated an inference speed of 0.069 s and an accuracy of 93.83% based on 300 apple samples. The integrated system, which included the proposed rotation mechanism and multi-camera setup, took a total of 2.84 s to sort one apple. Our proposed system provided an efficient and precise solution for detecting defects on the entire surface of apples, improving the sorting process with high reliability.

Graph convolutional neural networks (GCNNs) have been successfully applied to a wide range of problems, including low-dimensional Euclidean structural domains representing images, videos, and speech and high-dimensional non-Euclidean domains, such as social networks and chemical molecular structures. However, in computer vision, the existing GCNNs are not provided with positional information to distinguish between graphs of new structures; therefore, the performance of the image classification domain represented by arbitrary graphs is significantly poor. In this work, we introduce how to initialize the positional information through a random walk algorithm and continuously learn the additional position-embedded information of various graph structures represented over the superpixel images we choose for efficiency. We call this method the graph convolutional network with learnable positional embedding applied on images (IMGCN-LPE). We apply IMGCN-LPE to three graph convolutional models (the Chebyshev graph convolutional network, graph convolutional network, and graph attention network) to validate performance on various benchmark image datasets. As a result, although not as impressive as convolutional neural networks, the proposed method outperforms various other conventional convolutional methods and demonstrates its effectiveness among the same tasks in the field of GCNNs.

Accurate inspection of subtle defects on apple surfaces is necessary in agricultural engineering. However, existing methods often rely on expensive equipment and encounter difficulty in detecting small defect areas effectively. To address this challenge, we introduce the Subtle Surface Defects in Apples (SSDA) dataset, a custom dataset specifically collected and annotated for two defect types: scratches and pest damage. For benchmarking, we propose a heatmap-based anchor-free (HAFREE) detector, a novel end-to-end object detection architecture designed to localize subtle defects on apple surfaces. Unlike prior methods that rely on anchor boxes, HAFREE employs a heatmap-based approach to represent defects as keypoints using two-dimensional Gaussian heatmaps. We introduce a multiscale feature fusion block, targeting small objects by incorporating local and global contextual information. To mitigate overfitting, we also implement a patch training strategy incorporating full images and cropped patches during training as a regularizer. The proposed method achieves a mAP50 of 50.05% on the SSDA dataset, outperforming one- and two-stage anchor-based detectors and previous anchor-free approaches. Code is available at: https://github.com/nbngochan/HAFREE.

The conventional process of generating medical radiology reports is labor-intensive and time-consuming, requiring radiologists to describe findings meticulously from imaging studies. This manual approach often causes undesirable delays in patient care. Despite advancements in computer vision and deep learning, developing an effective computer-aided solution to generate automated medical reports remains challenging. The recent advancements in deep learning technology, especially with the advent of contrastive learning, have shown significant performance in natural language supervision. However, their application to medical report generation, particularly in the domain of chest x-rays (CXR), has been limited due to the lack of large annotated datasets. Many studies have proposed multimodal contrastive learning schemes to address the data scarcity problem for natural images. However, none of these techniques have been efficiently explored in terms of medical report generation. This study addresses these challenges by proposing a dual contrastive learning network (DuCo-Net) containing backbone and augmented networks. The backbone network is trained on the original data, while the augmented network emphasizes cross-model augmentation learning in a unified framework. DuCo-Net enables two complementary learning mechanisms: intra-modal learning, where each network learns specialized features within its modality (either image or text), and inter-twc-wang-3538463 modal learning, which captures relationships between image and text modalities through a combined loss function. This dual learning approach leverages modified DenseNet121 and BioBERT models with advanced pooling techniques specifically tailored for handling medical data. Comprehensive evaluations on two publicly available datasets demonstrate that DuCo-Net significantly outperforms current benchmarks. On the Indiana University Chest X-rays dataset, our proposed methodology demonstrates significant improvements across standard metrics (BLEU-1: 0.50, ROUGE: 0.40, METEOR: 0.24, F1: 0.40). For the MIMIC-CXR dataset, the framework maintains robust performance (BLEU-1: 0.42, ROUGE: 0.34, METEOR: 0.20, F1: 0.34), representing substantial improvements over existing state-of-the-art approaches in medical report generation.

This study explores the integration of concept bottleneck models (CBMs) with knowledge distillation (KD) while preserving the locality characteristics of the CBM. Although KD proves effective in model compression, compressed models often lack interpretability in their decision-making process. We enhance comprehensive explainability by maintaining CBMs’ inherent interpretability through our novel approach to knowledge distillation. We introduce visual concept knowledge distillation (VICO-KD), which transfers both explicit and implicit visual concepts from the teacher to the student model while preserving the local interpretability of the CBM, enabling accurate classification and clear visualization of evidence. VICO-KD demonstrates superior performance on benchmark datasets compared to Vanilla-KD, ensuring the student model learns visual concepts while maintaining the local interpretation capabilities of the teacher CBM. Our methodology shows competitive performance against existing concept models, and the student model, trained via VICO-KD, exhibits enhanced performance compared to the teacher during interventions. This study highlights the effectiveness of combining a CBM with KD to improve both interpretability and explainability in compressed models while maintaining locality properties.