Gnana Praveen Rajasekhar

Gnana Praveen Rajasekhar

Auto-generated profile, based on publicly available information

Gnana Praveen Rajasekhar

Computer Research CRIM) Institute of Montreal Canada

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
4 January 2023 in IEEE Transactions on Biometrics, Behavior, and Identity Science
Verified
Journal Article
Closed access
RPR. Gnana Praveen
R. Gnana Praveen
PCPatrick Cardinal
Patrick Cardinal
EGEric Granger
Eric Granger
Automatic emotion recognition (ER) has recently gained lot of interest due to its potential in many real-world applications. In this context, multimodal approaches have been shown to improve performance (over unimodal approaches) by combining diverse and complementary sources of information, providing some robustness to noisy and missing modalities. In this paper, we focus on dimensional ER based on the fusion of facial and vocal modalities extracted from videos, where complementary audio-visual (A-V) relationships are explored to predict an individual’s emotional states in valence-arousal space. Most state-of-the-art fusion techniques rely on recurrent networks or conventional attention mechanisms that do not effectively leverage the complementary nature of A-V modalities. To address this problem, we introduce a joint cross-attentional model for A-V fusion that extracts the salient features across A-V modalities, that allows to effectively leverage the inter-modal relationships, while retaining the intra-modal relationships. In particular, it computes the cross-attention weights based on correlation between the joint feature representation and that of the individual modalities. By deploying the joint A-V feature representation into the cross-attention module, it helps to simultaneously leverage both the intra and inter modal relationships, thereby significantly improving the performance of the system over the vanilla cross-attention module. The effectiveness of our proposed approach is validated experimentally on challenging videos from the RECOLA and AffWild2 datasets. Results indicate that our joint cross-attentional A-V fusion model provides a cost-effective solution that can outperform state-of-the-art approaches, even when the modalities are noisy or absent. Code is available at https://github.com/praveena2j/Joint-Cross-Attention-for-Audio-Visual-Fusion.

3 July 2024 in IEEE Journal of Selected Topics in Signal Processing
Verified
Journal Article
Closed access
RPR. Gnana Praveen
R. Gnana Praveen
JAJahangir Alam
Jahangir Alam
Multimodal emotion recognition has immense potential for the comprehensive assessment of human emotions, utilizing multiple modalities that often exhibit complementary relationships. In video-based emotion recognition, audio and visual modalities have emerged as prominent contact-free channels, widely explored in existing literature. Current approaches typically employ cross-modal attention mechanisms between audio and visual modalities, assuming a constant state of complementarity. However, this assumption may not always hold true, as non-complementary relationships can also manifest, undermining the efficacy of cross-modal feature integration and thereby diminishing the quality of audio-visual feature representations. To tackle this problem, we introduce a novel Incongruity-Aware Cross-Attention (IACA) model, capable of harnessing the benefits of robust complementary relationships while efficiently managing non-complementary scenarios. Specifically, our approach incorporates a two-stage gating mechanism designed to adaptively select semantic features, thereby effectively capturing the inter-modal associations. Additionally, the proposed model demonstrates an ability to mitigate the adverse effects of severely corrupted or missing modalities. We rigorously evaluate the performance of the proposed model through extensive experiments conducted on the challenging RECOLA and Aff-Wild2 datasets. The results underscore the efficacy of our approach, as it outperforms state-of-the-art methods by adeptly capturing inter-modal relationships and minimizing the influence of missing or heavily corrupted modalities. Furthermore, we show that the proposed model is compatible with various cross-modal attention variants, consistently improving performance on both datasets.

22 November 2023
Verified
Book Chapter
Closed access
PDF file
GRGnana Praveen Rajasekhar
Gnana Praveen Rajasekhar
JAJahangir Alam
Jahangir Alam

14 April 2024 in arXiv
Verified
Preprint
Open access
PDF file
RPR. Gnana Praveen
R. Gnana Praveen
JAJahangir Alam
Jahangir Alam
Though multimodal emotion recognition has achieved significant progress over recent years, the potential of rich synergic relationships across the modalities is not fully exploited. In this paper, we introduce Recursive Joint Cross-Modal Attention (RJCMA) to effectively capture both intra- and inter-modal relationships across audio, visual, and text modalities for dimensional emotion recognition. In particular, we compute the attention weights based on cross-correlation between the joint audio-visual-text feature representations and the feature representations of individual modalities to simultaneously capture intra- and intermodal relationships across the modalities. The attended features of the individual modalities are again fed as input to the fusion model in a recursive mechanism to obtain more refined feature representations. We have also explored Temporal Convolutional Networks (TCNs) to improve the temporal modeling of the feature representations of individual modalities. Extensive experiments are conducted to evaluate the performance of the proposed fusion model on the challenging Affwild2 dataset. By effectively capturing the synergic intra- and inter-modal relationships across audio, visual, and text modalities, the proposed fusion model achieves a Concordance Correlation Coefficient (CCC) of 0.585 (0.542) and 0.674 (0.619) for valence and arousal respectively on the validation set(test set). This shows a significant improvement over the baseline of 0.240 (0.211) and 0.200 (0.191) for valence and arousal, respectively, in the validation set (test set), achieving second place in the valence-arousal challenge of the 6th Affective Behavior Analysis in-the-Wild (ABAW) competition.

28 September 2023 in arXiv
Verified
Preprint
Open access
PDF file
RPR. Gnana Praveen
R. Gnana Praveen
JAJahangir Alam
Jahangir Alam
Speaker verification has been widely explored using speech signals, which has shown significant improvement using deep models. Recently, there has been a surge in exploring faces and voices as they can offer more complementary and comprehensive information than relying only on a single modality of speech signals. Though current methods in the literature on the fusion of faces and voices have shown improvement over that of individual face or voice modalities, the potential of audio-visual fusion is not fully explored for speaker verification. Most of the existing methods based on audio-visual fusion either rely on score-level fusion or simple feature concatenation. In this work, we have explored cross-modal joint attention to fully leverage the inter-modal complementary information and the intra-modal information for speaker verification. Specifically, we estimate the cross-attention weights based on the correlation between the joint feature presentation and that of the individual feature representations in order to effectively capture both intra-modal as well inter-modal relationships among the faces and voices. We have shown that efficiently leveraging the intra- and inter-modal relationships significantly improves the performance of audio-visual fusion for speaker verification. The performance of the proposed approach has been evaluated on the Voxceleb1 dataset. Results show that the proposed approach can significantly outperform the state-of-the-art methods of audio-visual fusion for speaker verification.

6 November 2024 in TechRxiv
Verified
Preprint
Open access
RPR Gnana Praveen
R Gnana Praveen
PCPatrick Cardinal
Patrick Cardinal
EGEric Granger
Eric Granger
Given the recent advancements in deep learning (DL), and computing capabilities, there has been significant progress in the development of systems analyzing facial behavior, evolving from images under controlled laboratory conditions, to more challenging real-world scenarios, including video-based recognition. However, DL models typically require training with large-scale datasets to provide a high level of performance. The collection and annotation of such datasets is a costly undertaking that relies on domain experts. Moreover, the annotation process is highly vulnerable to the ambiguity of expressions or Action Units (AUs) due to the bias induced by experts. This paper provides a comprehensive review of state-of-the-art weakly supervised learning (WSL) methods for facial behavior analysis based on images and videos. First, we introduce a taxonomy of relevant WSL scenarios, and then methods proposed for classification applications (using weak categorical/discrete labels), and for regression applications (using weak ordinal/dimensional labels). For both types of recognition applications (classification and regression), we provide a systematic review of state-of-the-art machine learning (ML) models for expression and AU recognition, and intensity estimation in each WSL scenario, discussing their strengths and limitations. An overview of widely used experimental methodologies (public datasets and protocols) is also provided for the performance evaluation of these state-of-the-art models. Finally, our critical analysis of the methods, along with their experimental results, provides insight into the key challenges of WSL in different scenarios, motivating future research to leverage weakly labeled data for real-world facial behavior analysis problems. Our review indicates that WSL methods may provide a cost-effective approach to train robust ML/DL models for facial behavior analysis in real-world scenarios. A website including constantlyupdated survey is provided at https://github.com/praveena2j/ awesome-Weakly-Supervised-Facial-Behavior-Analysis.