Approximating the uncertainty of an emotional AI agent is crucial for improving the reliability of such agents and facilitating human-in-the-loop solutions, especially in critical scenarios. However, none of the existing systems for emotion recognition in conversation (ERC) has attempted to estimate the uncertainty of their predictions. In this article, we present HU-Dialogue, which models hierarchical uncertainty for the ERC task. We perturb contextual attention weight values with source-adaptive noises within each modality, as a regularization scheme to model context-level uncertainty and adapt the Bayesian deep learning method to the capsule-based prediction layer to model modality-level uncertainty. Furthermore, a weight-sharing triplet structure with conditional layer normalization is introduced to detect both invariance and equivariance among modalities for ERC. We provide a detailed empirical analysis for extensive experiments, which shows that our model outperforms previous state-of-the-art methods on three popular multimodal ERC datasets.

Sentiment analysis is an important topic concerning identification of feelings, attitudes, emotions and opinions from text. A critical challenge for automating such analysis is the high manual annotation cost when conducting large-scale learning. However, the cross-domain technique is a key solution for this. It reuses annotated reviews across domains and its success principally relies on the learning of a robust common representation space across domains to support the sentiment classifier transfer. In the recent years, significant effort has been invested to improve the cross-domain representation learning by designing increasingly more complex and elaborate model inputs and architectures. We support that it is not necessary to focus on design complexity as this inevitably consumes more time for model training. Instead, we propose to explore through a simple mapping the word polarity and occurrence information and encode such information more accurately whilst aiming at lower computational costs. The proposed approach is unique and takes advantage of the stochastic embedding technique to tackle cross-domain sentiment alignment. Its effectiveness is benchmarked with over ten data tasks constructed from two review corpora, and is compared against ten classical and state-of-the-art methods.

Cross-modal hashing is an effective approach for information retrieval from large and heterogeneous cross-modal datasets, owing to its low storage cost and high computational speed. However, conventional cross-modal hashing techniques for generating hashing codes rely on cross-space dimensional compression, which results in two types of information loss: quantization information loss and dimension reduction loss. To address these limitations, we propose a novel method that decouples the one-step hashing (Fig.1a) strategy into two sub-steps (Fig.1b). Specifically, in the first step, we introduce a novel differentiable hash method, which utilizes a smooth hash module for binary quantization. This method allows our model to reduce the quantization information loss and make the model optimized by gradient descent. In the second step, we design a long-short Hamming space transformation approach to project the long code into a short one, which is effective in preserving the dimension information between long and short and mitigating the dimension reduction loss. We demonstrate the effectiveness of our approach through extensive experiments on several popular cross-modal datasets, achieving a significant improvement in cross-modal retrieval performance.

Recently, the rapid development of 2-dimensional (2D) convolutional neural networks (CNNs) has driven single image super-resolution (SISR) into a new era, owing to their powerful ability in modeling spatial relation within one single image. However, few studies focus on video super-resolution (VSR) due to the key challenge that apart from the spatial relation, the temporal dependence among consecutive low-resolution (LR) frames must be taken into consideration for better reconstruction. In this paper, unlike most previous methods based on optical flow for motion compensation, 3-dimensional (3D) convolution is utilized to capture the temporal relation. Firstly, in contrast to the conventional 3D convolution which is notorious for the excessively high computational burden, we propose an efficient 3D convolutional block (E3DB) through convolution factorization principle (CFP), which significantly reduces the computing load while maximally maintaining the temporal information. Then, by taking advantage of E3DB, we propose a novel multi-resolution extraction block (MREB) which aggregates the information from multiple resolutions, leading to the stronger high-resolution representation learning and better feature extraction. Besides, based on our 3-branch architecture, instead of the simple addition or concatenation, a dynamic reconstruction strategy (DRS) is proposed to adaptively fuse the optimal information of temporal dependence from each branch. It is therefore termed as dynamic multiple branch network (DMBN). Comprehensive experiments on public benchmark datasets demonstrate the superiority of our DMBN over the current state-of-the-art methods in terms of accuracy and efficiency.

The development of speech synthesis technology has increased the attention toward the threat of spoofed speech. Although various high-performance spoofing countermeasures have been proposed in recent years, a particular scenario is overlooked: partially spoofed audio, where spoofed utterances may contain both spoofed and bona fide segments. Currently, the research on partially spoofed speech detection is lacking. The existing methods either train with partially spoofed speech at utterance level, resulting in gradient conflicting at the segment level, or directly train with segment level data, which requires segment labels that are difficult to obtain in practice. In this study, to better detect partially spoofed speech when only utterance labels are available, we formulate partially spoofed speech detection into a multiple instance learning (MIL) problem. The typical MIL uses a pooling layer to fuse patch scores as a whole, and we propose a hybrid MIL (H-MIL) framework based on max and log-sum-exp pooling methods, which can learn better segment representations to improve partially spoofed speech detection performance. Theoretical and experimental verification shows that H-MIL can effectively relieve the gradient conflicting and gradient vanishing problems. In addition, we analyze the local correlations between segments and introduce a local self-attention mechanism to enhance segment features, which further promotes the detection performance. In our experiments, we provide not only detection results at the segment and utterance levels but also some detailed visualization analysis, including the effect of spoof ratio and cross-dataset detection. The experimental results demonstrate the effective detection performance of our method at both the utterance and segment levels, especially when dealing with low spoof ratio attacks. The results confirm that our approach can better deal with partially spoofed speech detection than previous methods.

Cross-modal hashing is a highly effective technique for searching relevant data across different modalities, owing to its low storage costs and fast similarity retrieval capability. While significant progress has been achieved in this area, prior investigations predominantly concentrate on a one-to-one feature alignment approach, where a singular feature is derived for similarity retrieval. However, the singular feature in these methods fails to adequately capture the varied multi-instance information inherent in the original data across disparate modalities. Consequently, the conventional one-to-one methodology is plagued by a semantic mismatch issue, as the rigid one-to-one alignment inhibits effective multi-instance matching. To address this issue, we propose a novel Diverse Instances Matching for Cross-modal Hashing (DIMCH), which explores the relevance between multiple instances in different modalities using a multi-instance learning algorithm. Specifically, we design a novel diverse instances learning module to extract a multi-feature set, which enables our model to capture detailed multi-instance semantics. To evaluate the similarity between two multi-feature sets, we adopt the smooth chamfer distance function, which enables our model to incorporate the conventional similarity retrieval structure. Moreover, to sufficiently exploit the supervised information from the semantic label, we adopt the weight cosine triplet loss as the objective function, which incorporates the multilevel similarity among the multi-labels into the training procedure and enables the model to mine the multi-label correlation effectively. Extensive experiments demonstrate that our diverse hashing embedding method achieves state-of-the-art performance in supervised cross-modal hashing retrieval tasks.