13 August 2024 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Understanding emotions from diverse contexts has received widespread attention in computer vision communities. The core philosophy of Context-Aware Emotion Recognition (CAER) is to provide valuable semantic cues for recognizing the emotions of target persons by leveraging rich contextual information. Current approaches invariably focus on designing sophisticated structures to extract perceptually critical representations from contexts. Nevertheless, a long-neglected dilemma is that a severe context bias in existing datasets results in an unbalanced distribution of emotional states among different contexts, causing biased visual representation learning. From a causal demystification perspective, the harmful bias is identified as a confounder that misleads existing models to learn spurious correlations based on likelihood estimation, limiting the models’ performance. To address the issue, we embrace causal inference to disentangle the models from the impact of such bias, and formulate the causalities among variables in the CAER task via a customized causal graph. Subsequently, we present a Contextual Causal Intervention Module (CCIM) to de-confound the confounder, which is built upon backdoor adjustment theory to facilitate seeking approximate causal effects during model training. As a plug-and-play component, CCIM can easily integrate with existing approaches and bring significant improvements. Systematic experiments on three datasets demonstrate the effectiveness of our CCIM.
19 January 2024 in IEEE Signal Processing Letters
The segmentation of liver tumors using multi-phase computed tomography (CT) images has garnered considerable attention in medical signal processing. However, existing multi-phase liver tumor segmentation methods primarily concentrate on feature integration across various phases, neglecting a comprehensive exploration of synergistic relationships among these phases and constraints on features across different scales. This limitation has led to performance bottlenecks in existing approaches. This article proposes a robust multi-phase liver tumor segmentation framework designed to address the aforementioned challenges. Specifically, we introduce a novel multi-phase and channel-stacked dual attention module, seamlessly integrated within a multi-scale architecture. This module adaptively captures essential semantic information among different phases, enhancing the segmentation network's feature extraction capabilities. A scale-weighted loss function for multi-scale supervision is also designed to mitigate false positives in the segmentation results. To facilitate a systematic evaluation of our model's performance on multi-phase data, we curate a new dataset comprising samples from four distinct phases. Our proposed framework is rigorously assessed through comprehensive quantitative and qualitative experiments, highlighting its compelling performance.
4 June 2024 in IEEE Signal Processing Letters
The signals from human expressions are usually multimodal, including natural language, facial gestures, and acoustic behaviors. A key challenge is how to fuse multimodal time-series signals with temporal asynchrony. To this end, we present a Transformer-driven Signal Interaction and Fusion (TSIF) approach to effectively model asynchronous multimodal signal sequences. TSIF consists of linear and cross-modal transformer modules with different duties. The linear transformer module efficiently performs the global interaction for multimodal signals, and the vital philosophy is to replace the dot product similarity with the Exponential Kernel while achieving linear complexity by a low-rank matrix decomposition. By targeting the language modality, the cross-modal transformer module aims to capture reliable element correlations among distinct signals and mitigate noise interference in audio and visual modalities. Numerous experiments on two multimodal benchmarks show that our TSIF comparably outperforms previous state-of-the-art models with lower space-time complexities. The systematic analysis also proves the effectiveness of the proposed modules.
23 September 2022 in Computational Intelligence and Neuroscience
Preoperative observation of liver status in patients with liver tumors by abdominal Computed Tomography (CT) imaging is one of the essential references for formulating surgical plans. Preoperative vessel segmentation in the patient’s liver region has become an increasingly important and challenging problem. Almost all existing methods first segment arterial and venous vessels on CT in the arterial and venous phases, respectively. Then, the two are directly registered to complete the reconstruction of the vascular system, ignoring the displacement and deformation of blood vessels caused by changes in body position and respiration in the two phases. We propose an unsupervised domain-adaptive two-stage vessel segmentation framework for simultaneous fine segmentation of arterial and venous vessels on venous phase CT. Specifically, we first achieve domain adaptation for arterial and venous phase CT using a modified cycle-consistent adversarial network. The newly added discriminator can improve the ability to generate and discriminate tiny blood vessels, making the domain-adaptive network more robust. The second-stage supervised training of arterial vessels was then performed on the translated arterial phase CT. In this process, we propose an orthogonal depth projection loss function to enhance the representation ability of the 3D U-shape segmentation network for the geometric information of the vessel model. The segmented venous vessels were also performed on venous phase CT in the second stage. Finally, we invited professional doctors to annotate arterial and venous vessels on the venous phase CT of the test set. The experimental results show that the segmentation accuracy of arterial and venous vessels on venous phase CT is 0.8454 and 0.8087, respectively. Our proposed framework can simultaneously achieve supervised segmentation of venous vessels and unsupervised segmentation of arterial vessels on venous phase CT. Our approach can be extended to other fields of medical segmentation, such as unsupervised domain adaptive segmentation of liver tumors at different CT phases, to facilitate the development of the community.