Multimodal sentiment analysis is a challenging task in the field of natural language processing (NLP). It uses multimodal signals (natural language, facial gestures, and acoustic behavior) in videos to generate emotional understanding. However, the importance of single modality data in the video to emotional outcomes is not static. With the extension of the time dimension, the emotional attributes of a specific natural language will be affected by non-natural language data, resulting in a vector shift in the feature space. At the same time, long-term dependencies within a specific modality and long-term dependencies between multiple modalities that are “unaligned” need to be considered. In response to the above problems, this paper proposes Multimodal Encoding-Decoding Network with Transformer. The network model encodes multimodal data through a Bidirectional Encoder Representations from Transformers (BERT) network and Transformer encoder to resolve long-term dependencies within modalities. And the network reconstructs the Transformer decoder to solve the weight problem of multimodal data in an iterative way. The network fully considers the long-term dependencies between modalities and the offset effect of non-natural language data on natural language data. Under the same experimental conditions, we validated our model on general multimodal sentiment analysis datasets. Compared with state-of-the-art models, the network achieves good progress and strong stability.

Multimodal sentiment analysis and emotion recognition represent a major research direction in natural language processing (NLP). With the rapid development of online media, people often express their emotions on a topic in the form of video, and the signals it transmits are multimodal, including language, visual, and audio. Therefore, the traditional unimodal sentiment analysis method is no longer applicable, which requires the establishment of a fusion model of multimodal information to obtain sentiment understanding. In previous studies, scholars used the feature vector cascade method when fusing multimodal data at each time step in the middle layer. This method puts each modal information in the same position and does not distinguish between strong modal information and weak modal information among multiple modalities. At the same time, this method does not pay attention to the embedding characteristics of multimodal signals across the time dimension. In response to the above problems, this paper proposes a new method and model for processing multimodal signals, which takes into account the delay and hysteresis characteristics of multimodal signals across the time dimension. The purpose is to obtain a multimodal fusion feature emotion analysis representation. We evaluate our method on the multimodal sentiment analysis benchmark dataset CMU Multimodal Opinion Sentiment and Emotion Intensity Corpus (CMU-MOSEI). We compare our proposed method with the state-of-the-art model and show excellent results.

Surface defects in hot-rolled steel strips are one of the common product problems for the steel industry, which harm the product appearance, affect the corrosion and wear resistance, and shorten the product service life. The natural defect samples are sparse, category imbalanced, and expensive manual annotations. Therefore, it is crucial to study the data augmentation and classification methods for small sample surface defects. To solve the above problems and improve the accuracy and real-time performance of defect classification, we propose a random offline data augmentation algorithm (Random-CutMix) and an improved MobileNet architecture (SP-MobileNet). The Random-CutMix algorithm expands the dataset by random sampling to balance the number of each defect class. The SP-MobileNet combines the inverse residual module with the channel shuffle mechanism (CSIn-Module) and pyramid split attention (PSA) module, which facilitates cross-group information flow and improves model representation capability and generalization performance with low computational cost. The accuracy, recall, F1 score, parameter, computational complexity, and frame rate of SP-MobileNet with Random-CutMix on the X-SDD dataset were 95.97%, 95.22%, 95.46%, 6.5 M, and 0.54 G, 72 FPS, respectively. The experiment results indicate that our method outperforms the state-of-the-art methods and provides an effective trade-off between accuracy and instantaneity in actual industrial production.

The enrichment of social media expression makes multimodal sentiment analysis a research hotspot. However, modality heterogeneity brings great difficulties to effective cross-modal fusion, especially the modality alignment problem and the uncontrolled vector offset during fusion. In this paper, we propose a bimodal multi-head attention network (BMAN) based on text and audio, which adaptively captures the intramodal utterance features and complex intermodal alignment relationships. Specifically, we first set two independent unimodal encoders to extract the semantic features within each modality. Considering that different modalities deserve different weights, we further built a joint decoder to fuse the audio information into the text representation, based on learnable weights to avoid an unreasonable vector offset. The obtained cross-modal representation is used to improve the sentiment prediction performance. Experiments on both the aligned and unaligned CMU-MOSEI datasets show that our model achieves better performance than multiple baselines, and it has outstanding advantages in solving the problem of cross-modal alignment.

Nowadays, defect detection technology based on deep learning continuously increases the surface quality requirements of hot-rolled strip steel. However, due to limitations in industrial production, defect datasets often suffer from insufficient training samples and imbalanced categories. This paper proposes effective solutions, namely the GT-CutMix offline data augmentation algorithm and lightweight small sample defect detection models. The GT-CutMix augmentation algorithm significantly improves defect utilization by accurately sampling defect locations and integrating them into the original data set. We design the S-deconvolutional single shot detector (DSSD) defect detection model by constructing a lightweight SI-MobileNet to replace the ResNet101 backbone of the DSSD network. This can reduce the resource parameters and consumption. At the same time, it can speed up training and inference. To further improve the detection accuracy, we integrate the pyramid split attention (PSA) mechanism into the prediction module of DSSD and construct the SA-DSSD model. Under the GT-CutMix augmentation algorithm, the mAP of S-DSSD and SA-DSSD models on X-SDD dataset are 76.83% and 78.63%, respectively. Meanwhile, the corresponding detection speeds are 45 FPS and 40 FPS, respectively. In addition, on the NEU-DET cross-dataset experiment, the mAP of the SA-DSSD model reaches 74.88%. Our methods are highly effective and generalized for small sample defect detection, which can provide selective solutions for specific needs such as high speed and precision in different industrial production scenarios.

Industrial defect detection plays a critical role in controlling product quality. Obtaining industrial defects with diverse and balanced classes in natural environments is often challenging. Most methods tend to uniformly augment all classes in small-sample datasets, which wastes computing resources and the classification performance is not always good. To achieve the purposive data augmentation, we propose a minority class imbalance rate (MiCIR) and an MiCIR-based data augmentation strategy that can determine the class and the number of samples to be augmented. In addition, to address the misclassification problem of classes with relatively large sample sizes, we introduce a lightweight classification model, ShcNet. We construct convolution-batchnorm-hard-swish (CBH) and convolution-batchnorm-hard-swish-convolutional block attention mechanism (CBHC) modules in ShcNet to improve classification performance. Experimental results demonstrate that our data augmentation strategy can significantly improve the classification results with generalizability across different datasets. The ShcNet outperforms the baseline models on classification accuracy while maintaining fewer parameters and model complexity.