Visual dialog is a challenging vision-language task in which a series of questions visually grounded by a given image are answered. To resolve the visual dialog task, a high-level understanding of various multimodal inputs (e.g., question, dialog history, and image) is required. Specifically, it is necessary for an agent to (1) determine the semantic intent of question and (2) align question-relevant textual and visual contents among heterogeneous modality inputs. In this paper, we propose Multi-View Attention Network (MVAN), which leverages multiple views about heterogeneous inputs based on attention mechanisms. MVAN effectively captures the question-relevant information from the dialog history with two complementary modules (i.e., Topic Aggregation and Context Matching), and builds multimodal representations through sequential alignment processes (i.e., Modality Alignment). Experimental results on VisDial v1.0 dataset show the effectiveness of our proposed model, which outperforms previous state-of-the-art methods under both single model and ensemble settings.

The use of sufficiently large datasets is important for most deep learning tasks, and emotion recognition tasks are no exception. Multimodal emotion recognition is the task of considering multiple types of modalities simultaneously to improve accuracy and robustness, typically utilizing three modalities: visual, audio, and text. Similar to other deep learning tasks, large datasets are required. Various heterogeneous datasets exist, including unimodal datasets constructed for traditional unimodal recognition and bimodal or trimodal datasets for multi-modal emotion recognition. A trimodal emotion recognition model shows high performance and robustness by comprehensively considering multiple modalities. However, the use of unimodal or bimodal datasets in this case is problematic. In this study, we propose a novel method to improve the performance of emotion recognition based on a cross-modal translator that can translate between the three modalities. The proposed method can train a multimodal model based on three modalities with different types of heterogeneous datasets, and the dataset does not require alignment between modalities: visual, audio, and text. We achieved a high performance exceeding the baseline in CMU-MOSEI and IEMOCAP, which are representative multimodal datasets, by adding unimodal and bimodal datasets to the trimodal dataset.

Image captioning has received significant interest in recent years, and notable results have been achieved. Most previous approaches have focused on generating visual descriptions from images, whereas a few approaches have exploited visual descriptions for image classification. This study demonstrates that a good performance can be achieved for both description generation and image classification through an end‐to‐end joint learning approach with a loss function, which encourages each task to reach a consensus. When given images and visual descriptions, the proposed model learns a multimodal intermediate embedding, which can represent both the textual and visual characteristics of an object. The performance can be improved for both tasks by sharing the multimodal embedding. Through a novel loss function based on class activation mapping, which localizes the discriminative image region of a model, we achieve a higher score when the captioning and classification model reaches a consensus on the key parts of the object. Using the proposed model, we established a substantially improved performance for each task on the UCSD Birds and Oxford Flowers datasets.

In recent years, emotion recognition has become an increasingly vital tool for enhancing customer service applications. Especially in telephonic interactions, detecting emotions accurately is crucial for improving human-computer interaction experiences. Despite significant advances in deep learning, current emotion recognition systems that integrate voice and text face challenges such as noise interference in transcripts and inadequate multimodal fusion, which hinder precise emotion detection. In this paper, we introduce MASKSER, a methodology that combines vocal signals and transcribed text in a robust manner. Our approach involves pretraining noisy transcripts with ChatGPT-4 using few-shot learning based on techniques such as masking and sentiment word replacement. This enhances emotion discernment significantly by leveraging the strengths of both modalities. To address the challenges posed by noisy data, we propose a mask-based noise generation model and use it to pretrain the transcript-based model, which helps mitigate inaccuracies. Additionally, we introduce a novel loss function that evaluates the Kullback-Leibler divergence between text and voice encoder distributions, ensuring balanced contributions from both modalities. Experiments are conducted in both English and Korean to validate the language independence and robustness of the proposed approach in different linguistic contexts. The results demonstrate substantial improvements in emotion recognition capabilities, achieving high performance metrics while reducing reliance on costly speech recognition resources.

Open-world object detection is a challenging set of tasks in the realm of computer vision. In these tasks, the object detection model processes the input image or video and undergoes inference to identify objects or features present in the image or video. The main objective of the model is to identify the seen and unseen classes rather than identifying only the classes that that were introduced to it during training. The challenging task is to create instance-level true bounding boxes tightly around the true objects and classify and localize them with their true class labels, without missing any of the true object or assigning false positive labels to them. Motivated by the pressing need to advance the capabilities of open-world object detection we present a novel clustering technique called margin-based latent space clustering and deployed it in the classification head of the Faster RCNN. Furthermore, we propose a novel loss function called margin-based loss coupled with regularization parameters aiming to optimize the outlier identification. The proposed method is also capable of incremental learning. Overall, all four incremental tasks are assessed by employing benchmark evaluation metrics. The proposed method outperforms the existing state-of-the-art method, with significant improvements in mean average precision (mAP). We improved the mAP by 4.49% on task 1, 3.71% on task 2, 6.19% on task 3 and 2.81% on task 4. We also reduced the ‘Wilderness Impact and Absolute Open-Set Error’ metrics (For assessing the false positive detection, both metrics are employed), on every task.