The Multimodal Sentiment Analysis Challenge presents two distinct sub-challenges related to human perception characteristics. This paper focuses on the MUSE-PERCEPTION challenge, which aims to predict the perceptual attributes of CEOs from video data. Specifically, we propose a novel approach that leverages Multimodal Large Language Models (MM-LLM) to integrate multimodal features and generate accurate predictions. Our proposal explores both the direct generation of target values using tuned Qwen-VL and the extraction of semantic representations from a pre-trained Gemma-2B model. The latter integrates multimodal data such as facial expressions (vit-fer), audio (w2v-msp) and additional features such as egemaps or facenet512 results. The input data involve a task-specific cue and an audio transcript. Feature projectors map each modality into the semantic space of the LLM, and mean or attention clustering aggregates the final hidden state of the LLM. Next, an MLP uses these aggregated representations to predict 16 perceptual attributes. To optimize model performance, we experimented with several fitting methods and implemented a combined mean absolute error (MAE) and Pearson correlation coefficient (ρ) loss function. The experimental results shows the effectiveness of this approach, and the Gemma-2B-based model achieves a solid second place in the challenge. These results underscore the potential of MM-LLMs to improve the understanding of human perception in professional contexts.

Designing reliable automatic models for social perception can contribute to a better understanding of human behavior, enabling more trustworthy experiences in the multimedia on-line communication environment. However, predicting social attributes from video data remains challenging due to the complex interplay of visual, auditory, and linguistic cues. In this paper, we address this challenge by investigating the effectiveness of Multimodal Large Language Models (MM-LLMs) for feature extraction in the MuSe-Perception challenge. Firstly, our analysis of the novel LMU-ELP dataset has revealed high correlations between certain perceptual dimensions, motivating using a single regression model for all 16 social attributes to be predicted for a set of speakers appearing in recorded video clips. We demonstrate that dimensionality reduction through Principal Component Analysis (PCA) can be applied to the label space without a relevant performance loss. Secondly, by employing frozen MM-LLMs as feature extractors, we explore their ability to capture perception-related information. We extract sequence embeddings from the Qwen-VL and Qwen-Audio models and train a Multi-Layer Perceptron over the attention-pooled vectors for each one of the encoders, obtaining a mean Pearson correlation of 0.22 using the average predictions for both models. Our best result of 0.31 is achieved by training the same architecture over the baseline vit-ver and w2v-msp features, which motivates further exploration on how to effectively leverage advanced MM-LLMs as feature extractors. Lastly, a post hoc analysis of our results highlights the limitations of Pearson correlation for evaluating regression performance in this context. In particular, a similar Pearson coefficient can be obtained with two very different prediction sets displaying different levels of variability. We take this result as a call to action in exploring alternative metrics to assess the regression performance for the task.

The accurate modelling of video memorability, or the intrinsic properties that render a piece of audiovisual content more likely to be remembered, will facilitate the development of automatic systems that are more efficient in retrieving, classifying and generating impactful media. Recent studies have indicated a strong correlation between the visual semantics of video and its memorability. This underscores the importance of developing advanced visual comprehension abilities to enhance model performance. It has been demonstrated that Large Vision–Language Models (LVLMs) demonstrate exceptional proficiency in generalist, high-level semantic comprehension of images and video, due to their extensive multimodal pre-training on a vast scale. This work makes use of the vast generalist knowledge of LVLMs and explores efficient adaptation techniques with a view to utilising them as memorability predictors. In particular, the Quantized Low-Rank Adaptation (QLoRA) technique is employed to fine-tune the Qwen-VL model with memorability-related data extracted from the Memento10k dataset. In light of existing research, we propose a particular methodology that transforms Qwen-VL from a language model to a memorability score regressor. Furthermore, we consider the influence of selecting appropriate LoRA hyperparameters, a design aspect that has been insufficiently studied. We validate the LoRA rank and alpha hyperparameters using 5-Fold Cross-Validation and evaluate our best configuration on the official testing portion of the Memento10k dataset, obtaining a state-of-the-art Spearman Rank Correlation Coefficient (SRCC) of 0.744. Consequently, this work represents a significant advancement in modelling video memorability through high-level semantic understanding.

The memorability of a video is defined as an intrinsic property of its visual features that dictates the fraction of people who recall having watched it on a second viewing within a memory game. Still, unravelling what are the key features to predict memorability remains an obscure matter. This challenge is addressed here by fine-tuning text and image encoders using a cross-modal strategy known as Contrastive Language-Image Pre-training (CLIP). The resulting video-level data representations learned include semantics and topic-descriptive information as observed from both modalities, hence enhancing the predictive power of our algorithms. Our proposal achieves in the text domain a significantly greater Spearman Rank Correlation Coefficient (SRCC) than a default pre-trained text encoder (0.575 ± 0.007 and 0.538 ± 0.007, respectively) over the Memento10K dataset. A similar trend, although less pronounced, can be noticed in the visual domain. We believe these findings signal the potential benefits that cross-modal predictive systems can extract from being fine-tuned to the specific issue of media memorability.

Olfactory information, crucial for human perception, is often underrepresented compared to visual and textual data. This work explores methods for understanding smell descriptions within a multimodal context, where scent information is conveyed indirectly through text and images. We address the challenges of the Multimodal Understanding of Smells in Texts and Images (MUSTI) task by proposing novel approaches that leverage language-specific models and state-of-the-art multimodal large language models (MM-LLMs). Our core contribution is a multimodal framework using language-specific encoders for text and image data. This allows for a joint embedding space that explores the semantic symmetry between smells, texts, and images to identify olfactory-related connections shared across the modalities. While ensemble learning with language-specific models achieved good performance, MM-LLMs demonstrated exceptional potential. Fine-tuning a quantized version of the Qwen-VL-Chat model achieved a state-of-the-art macro F1-score of 0.7618 on the MUSTI task. This highlights the effectiveness of MM-LLMs in capturing task requirements and adapting to specific formats.

A detailed analysis of the inertial signals input is required when using deep learning models for Parkinson’s Disease detection. This work explores the possibility of reducing the input size of the models by studying the most appropriate frequency range and determines if it is feasible to evaluate subjects with different sensor locations than those used during training. For experimentation, 3.2 s windows are used to classify signals between Parkinson’s patients and control subjects, applying Fast Fourier Transform to the inertial signals and following a Leave-One-Subject-Out Cross-Validation methodology for the PD-BioStampRC21 dataset. It has been observed that the frequency range of 0 to 5 Hz offers a classification accuracy rate of 75.75 ± 0.62% using the five available sensors for training and evaluation, which is close to the model’s performance over the entire frequency range, from 0 to 15.625 Hz, which is 77.46 ± 0.60%. Regarding the transfer learning between sensors located in different body parts, it was observed that training and evaluating the model using data from the right forearm resulted in an accuracy of 65.17 ± 0.69%. When the model was trained with data from the opposite forearm, the accuracy was similar, at 63.57 ± 0.69%. Likewise, comparable results were found when using data from the other forearm and when training and evaluating with opposite thighs, with accuracy reductions not exceeding 3%.