Automatic gesture synthesis from speech is a topic that has attracted researchers for applications in remote communication, video games and Metaverse. Learning the mapping between speech and 3D full-body gestures is difficult due to the stochastic nature of the problem and the lack of a rich cross-modal dataset that is needed for training. In this paper, we propose a novel transformer-based framework for automatic 3D body gesture synthesis from speech. To learn the stochastic nature of the body gesture during speech, we propose a variational transformer to effectively model a probabilistic distribution over gestures, which can produce diverse gestures during inference. Furthermore, we introduce a mode positional embedding layer to capture the different motion speeds in different speaking modes. To cope with the scarcity of data, we design an intra-modal pre-training scheme that can learn the complex mapping between the speech and the 3D gesture from a limited amount of data. Our system is trained with either the Trinity speech-gesture dataset or the Talking With Hands 16.2M dataset. The results show that our system can produce more realistic, appropriate, and diverse body gestures compared to existing state-of-the-art approaches.

Hyperspectral imaging combined with chemometric approaches is proven to be a powerful tool for the quality evaluation and control of fruits. In fruit defect-detection scenarios, developing an unsupervised anomaly detection framework is vital, as defect sample preparation is labor-intensive and time-consuming, especially for exploring potential defects. In this paper, a spectral–spatial, information-based, self-supervised anomaly detection (SSAD) approach is proposed. During training, an auxiliary classifier is proposed to identify the projection axes of principal component (PC) images that were transformed from the hyperspectral data cubes. In test time, the fully connected layer of the learned classifier was used as a ‘spectral–spatial’ feature extractor, and the feature similarity metric was adopted as the score function for the downstream anomaly evaluation task. The proposed network was evaluated with two fruit data sets: a strawberry data set with bruised, infected, chilling-injured, and contaminated test samples and a blueberry data set with bruised, infected, chilling-injured, and wrinkled samples as anomalies. The results show that the SSAD yielded the best anomaly detection performance (AUC = 0.923 on average) over the baseline methods, and the visualization results further confirmed its advantage in extracting effective ‘spectral–spatial’ latent representation. Moreover, the robustness of SSAD is verified with the data pollution experiment; it performed significantly better than the baselines when a portion of anomalous samples was involved in the training process.

Conventional food fraud detection using hyperspectral imaging (HSI) relies on the discriminative power of machine learning. However, these approaches often assume a balanced class distribution in an ideal laboratory environment, which is impractical in real-world scenarios with diverse label distributions. This results in suboptimal performance when less frequent classes are overshadowed by the majority class during training. Thus, the critical research challenge emerges of how to develop an effective classifier on a small-scale imbalanced dataset without significant bias from the dominant class. In this paper, we propose a novel nondestructive detection approach, which we call the Dice Loss Improved Self-Supervised Learning-Based Prototypical Network (Proto-DS), designed to address this imbalanced learning challenge. The proposed amalgamation mitigates the label bias on the most frequent class, further improving robustness. We validate our proposed method on three collected hyperspectral food image datasets with varying degrees of data imbalance: Citri Reticulatae Pericarpium (Chenpi), Chinese herbs, and coffee beans. Comparisons with state-of-the-art imbalanced learning techniques, including the Synthetic Minority Oversampling Technique (SMOTE) and class-importance reweighting, reveal our method’s superiority. Notably, our experiments demonstrate that Proto-DS consistently outperforms conventional approaches, achieving the best average balanced accuracy of 88.18% across various training sample sizes, whereas the Logistic Model Tree (LMT), Multi-Layer Perceptron (MLP), and Convolutional Neural Network (CNN) approaches attain only 59.42%, 60.38%, and 66.34%, respectively. Overall, self-supervised learning is key to improving imbalanced learning performance and outperforms related approaches, while both prototypical networks and the Dice loss can further enhance classification performance. Intriguingly, self-supervised learning can provide complementary information to existing imbalanced learning approaches. Combining these approaches may serve as a potential solution for building effective models with limited training data.

In recent years, domain generalization-based fault diagnosis (DGFD) methods have shown significant potential in rotating machinery fault diagnosis in unseen target domains. However, these methods focus on learning domain-invariant representations via feature distribution adaptation. The generalization of classifiers and the orthogonality between fault-related and domain-related features have not been thoroughly explored, which hinders further improvements in DGFD performance. To address these limitations, an episodic training and feature orthogonality-driven domain generalization (EODG) method is proposed. In this method, episodic training is introduced to jointly improve the generalization capabilities of both the feature extractor and fault classifier, while a novel feature transfer loss is proposed for learning domain-invariant representations. Furthermore, the orthogonality between fault-related and domain-related features is enhanced by minimizing their cosine similarity, thereby improving the generalization capability of the DGFD model. The experimental results validated the effectiveness and superiority of the proposed method on domain generalization-based fault diagnosis tasks.