After its inception, emotion recognition or affective computing has increasingly become an active research topic due to its broad applications. The corresponding computational models have gradually migrated from statistically shallow models to neural-network-based deep models, which can significantly boost the performance of emotion recognition and consistently achieve the best results on different benchmarks, and thus has been considered the first option for emotion recognition. However, the debut of large language models (LLMs), such as ChatGPT and GPT4, has remarkably astonished the world due to their emerged capabilities of zero/few-shot learning, in-context learning (ICL), chain-of-thought, and others that are never shown in previous deep models. In the present article, we comprehensively investigate how the LLMs perform in emotion recognition in terms of diverse aspects, including ICL, few-shot prompting, accuracy, generalization, and explanation. Moreover, we offer some insights and pose other potential challenges, hoping to ignite broader discussions about enhancing emotion recognition in the new era of advanced and more generalized models.

Although recent research on speech emotion recognition has demonstrated that learning domain-invariant features provide an elegant solution to domain mismatch, the features learned by the existing methods lack generalization capabilities to capture latent information from datasets. We propose two novel domain adaptation methods, the generalized domain adversarial neural network (GDANN) and the class-aligned GDANN (CGDANN), to learn generalized domain-invariant representations for emotion recognition. GDANN and CGDANN, which are derived from multitask learning (MTL), consist of three tasks. The main task is to recognize the emotional category to which the input belongs. The remaining two tasks are auxiliary tasks. One is to use a variational autoencoder to model the input distribution, which encourages the model to learn the distribution of latent representations. The other is to learn the common representations of different domains, for which distinguishing via the domain classifier is difficult. The gradient of the domain classifier guides the shared representations of the source and target domains to approximate each other using a gradient reversal layer. To evaluate the effectiveness of the proposed methods, we conduct several experiments with the IEMOCAP and MSP-IMPROV datasets. The results illustrate that good performance is achieved compared with that of state-of-the-art methods. Notably, CGDANN utilizes a small quantity of labeled target domain samples to align the distribution representation and obtains the best performance among the comparison methods. We further visualize the representations learned by the proposed methods and discover that the representations of the source and target domains converge with a low variance.

Supervised cross-modal image-text hashing has aroused extensive concentrations in comprehending the correspondence between vision and language for data search tasks. Existing methods learn the compact hash codes by leveraging a given image-text data pairs or supervised information to explore such correspondence. However, they still confront obvious drawbacks. First, there is no engagement between multiple semantic information that yields the suboptimal search performance. Second, most of them adopt continuous relaxation strategy by discarding the discrete constraints, which results in large binary quantization errors. To deal with these problems, we propose a novel supervised hashing method, termed Discrete Joint Semantic Alignment Hashing (DJSAH). Specifically, it builds a connection between semantics (a.k.a. class labels and pairwise similarities) by the joint semantic alignment learning. And thus the high-level discriminative semantics can be preserved into the hash codes. Besides, a well-designed discrete optimization algorithm with linear computation and memory cost is developed to reduce the information loss of the hash codes with no need for relaxation. Extensive experiments and analyses on three benchmark datasets validate the superiority of the proposed DJSAH against several state-of-the-art hashing methods.

Despite the recent great achievements in speech emotion recognition (SER) with the development of deep learning, the performance of SER systems depends strongly on the amount of labeled data available for training. Obtaining sufficient annotated data, however, is often extremely time consuming and costly and sometimes even prohibitive because of privacy and ethical concerns. To address this issue, this article proposes the semisupervised generative adversarial network (SSGAN) for SER to capture underlying knowledge from both labeled and unlabeled data. The SSGAN is derived from a GAN, but the discriminator of the SSGAN can not only classify its input samples as real or fake but also distinguish their emotional class if they are real. Thus, the distribution of realistic inputs can be learned to encourage label information sharing between labeled and unlabeled data. This article proposes two advanced methods, i.e., the smoothed SSGAN (SSSGAN) and the virtual smoothed SSGAN (VSSSGAN), which, respectively, smooth the data distribution of the SSGAN via adversarial training (AT) and virtual adversarial training (VAT). The SSSGAN smooths the conditional label distribution given inputs using labeled examples, while the VSSSGAN smooths the conditional label distribution without label information (“virtual” labels). To evaluate the effectiveness of the proposed methods, four publicly available and frequently used corpora are selected to conduct experiments in intradomain and interdomain situations. The results illustrate that the proposed methods are superior to the state-of-the-art methods. Specifically, in experimental settings with mismatched and semimismatched unlabeled training sets, the SSSGAN and VSSSGAN are more robust than the SSGAN because of the distributional smoothness.