2 June 2023 in IEEE/ACM Transactions on Audio, Speech, and Language Processing
Aspect Sentiment Triplet Extraction (ASTE) is an emerging task of fine-grained sentiment analysis, which aims to extract aspect terms, associated opinion terms, and sentiment polarities in the form of triplets. Thus, ASTE involves two groups of subtasks: aspect/opinion term extraction and aspect-opinion-pair sentiment classification. Due to the high correlations of subtasks, three categories of joint methods have been proposed, including end-to-end tagging-based methods , cascaded span-based methods , and sequence-to-sequence generation-based methods . These methods basically learn either a shared feature space or a shared sentence encoder to capture interactions across all subtasks by parameter sharing. However, they fail to learn deep and mutual interactive features for ASTE. In this work, we present a novel tagging scheme to cast ASTE as a unified boundary-words relation classification problem. Subsequently, we propose an end-to-end Hierarchical Interaction Model (HIM), exploiting deep and mutual interactions across subtasks mainly with two interaction modules. The first-level interaction module primarily leverages multi-task learning models to capture implicit subtask interactions. Then, the second-level interaction module, namely Gated Interaction Network (GIN), adopts a novel gated control mechanism and a newly-designed Conditional BiLSTM (Cond-BiLSTM) network to capture explicit subtask interactions. Moreover, to refine the unreliable outputs of the first-level module, we develop a General word-Pair Relationship Learning (G-PRL) component. With the task-shared features as input, G-PRL further facilitates interactions between term extraction and pair classification. We conduct experiments on two benchmarks and achieve promising results. Extensive analyses demonstrate the effectiveness and flexibility of our work.
24 March 2024 in Proceedings of the AAAI Conference on Artificial Intelligence
This paper presents a new supervised representation learning framework, namely structured probabilistic coding (SPC), to learn compact and informative representations from input related to the target task. SPC is an encoder-only probabilistic coding technology with a structured regularization from the target space. It can enhance the generalization ability of pre-trained language models for better language understanding. Specifically, our probabilistic coding simultaneously performs information encoding and task prediction in one module to more fully utilize the effective information from input data. It uses variational inference in the output space to reduce randomness and uncertainty. Besides, to better control the learning process of probabilistic representations, a structured regularization is proposed to promote uniformity across classes in the latent space. With the regularization term, SPC can preserve the Gaussian structure of the latent code and achieve better coverage of the hidden space with class uniformly. Experimental results on 12 natural language understanding tasks demonstrate that our SPC effectively improves the performance of pre-trained language models for classification and regression. Extensive experiments show that SPC can enhance the generalization capability, robustness to label noise, and clustering quality of output representations.
22 July 2025 in IEEE Transactions on Multimedia
The rapid development of online media has heightened the importance of multimodal emotion recognition (MER) in video analysis. However, practical applications often encounter challenges due to missing modalities caused by various interferences. It is difficult to predict the specific missing situations, such as the number and types of missing modalities. Current approaches to modality missing typically apply a uniform method to address various missing cases, which are insufficiently adaptive to dynamic conditions. For example, translation-based methods can efficiently complete missing text from audio, but generating audio or video features that retain the original emotional information from other modalities is challenging and may introduce additional noise. In this paper, we introduce ROSA, a novel robust self-adaptive model designed to address various missing cases with tailored approaches, leveraging available modalities effectively and reducing the introduction of additional noise. Specifically, the A-T Completion module based on the encoder-decoder architecture enables ROSA to generate missing raw text from audio rather than mere embedding representations, capturing more nuanced modal features. Additionally, we design the T-V Fusion module based on a vision-language large model for deep extraction and fusion of textual and visual features. Comprehensive experiments conducted on three widely used public datasets demonstrate the superiority and effectiveness of our model. ROSA outperforms other models in both fixed missing rate and fixed missing modality cases. The ablation studies further highlights the contribution of each designed module.