In current multimodal sentiment analysis, aligned and complete multimodal sequences are often crucial. Obtaining complete multimodal data in the real world presents various challenges, and aligning multimodal sequences often requires a significant amount of effort. Unfortunately, most multimodal sentiment analysis methods fail when dealing with missing modalities or unaligned multimodal sequences. To tackle these two challenges simultaneously in a simple and lightweight manner, we present the Unified Multimodal Framework (UniMF). The primary components of UniMF comprise two distinct modules. The first module, Translation Module, translates missing modalities using information from existing modalities. The second module, Prediction Module, uses the attention mechanism to fuse the multimodal information and generate predictions. To enhance the translation performance of the Translation Module, we introduce the Multimodal Generation Mask (MGM) and utilize it to construct the Multimodal Generation Transformer (MGT). The MGT can generate the missing modality while focusing on information from existing modalities. Furthermore, we introduce the Multimodal Understanding Transformer (MUT) in the Prediction Module, which includes the Multimodal Understanding Mask (MUM) and a unique sequence, MultiModalSequence ( MMSeq ), representing a unified multimodality. To assess the performance of UniMF, we perform experiments on four multimodal sentiment datasets, and UniMF attains competitive or state-of-the-art outcomes with fewer learnable parameters. Furthermore, the experimental outcomes signify that UniMF, supported by MGT and MUT - two transformers utilizing special attention mechanisms, can efficiently manage both generating task of missing modalities and understanding task of unaligned multimodal sequences. The implementation of this work is publicly accessible at https://github.com/gw-zhong/UniMF .

In order to achieve better results of human complex activity recognition, the optimal feature representation of human complex activities from sensor data are studied. Features from multiple sets that are obtained by the learned networks, the manual extraction, and the location information are fused to generate mixed features to achieve better results of human complex activity recognition. A method extracting multi-layer features from the hybrid CNN and BLSTM network is proposed. The output feature vector of the second layer of the CNN and that of the BLSTM are combined to generate multi-layer features. Meanwhile, a new feature selection method based on SFS and network weight analysis is proposed. The method is to select a series of features extracted from the segmented sensor data using manual methods as dominant features. The location information of complex activities after one-hot encoding is also used as feature, which is then fused with multi-layer features and manual dominant features to generate mixed features. To further improve the recognition performance, the sensor data collected at different body positions are separated and used independently to train the hybrid CNN and BLSTM network to obtain their respective state information. The PAMAP2 and UT-Data datasets are used in our experiments to verify the proposed method. The results show that the method based on multiple features proposed in this paper dramatically outperforms the existing state-of-the-art methods for human complex activity recognition using sensor data.

Annotated electroencephalogram (EEG) data is the prerequisite for artificial intelligence-driven EEG autoanalysis. However, the scarcity of annotated data due to its high-cost and the resulted insufficient training limits the development of EEG autoanalysis. Generative self-supervised learning, represented by masked autoencoder, offers potential but struggles with non-Euclidean structures. To alleviate these challenges, this work proposes a self-supervised graph masked autoencoder for EEG representation learning, named GMAEEG. Concretely, a pretrained model is enriched with temporal and spatial representations through a masked signal reconstruction pretext task. A learnable dynamic adjacency matrix, initialized with prior knowledge, adapts to brain characteristics. Downstream tasks are achieved by finetuning pretrained parameters, with the adjacency matrix transferred based on task functional similarity. Experimental results demonstrate that with emotion recognition as the pretext task, GMAEEG reaches superior performance on various downstream tasks, including emotion, major depressive disorder, Parkinson's disease, and pain recognition. This study is the first to tailor the masked autoencoder specifically for EEG representation learning considering its non-Euclidean characteristics. Further, graph connection analysis based on GMAEEG may provide insights for future clinical studies.

Wearable electrocardiogram (ECG) monitoring devices have enabled everyday ECG collection in our daily lives. However, the condition of ECG signal acquisition using wearable devices varies and wearable ECG signals could be interfered with by severe noises, resulting in great challenges of computer-aided automated ECG analysis, especially for single-lead ECG signals without spare channels as references. There remains room for improvement of the beat-level single-lead ECG diagnosis regarding accuracy and efficiency. In this paper, we propose new morphological features of heartbeats for an extreme gradient boosting-based beat-level ECG analysis method to carry out the five-class heartbeat classification according to the Association for the Advancement of Medical Instrumentation standard. The MIT-BIH Arrhythmia Database (MITDB) and a self-collected wearable single-lead ECG dataset are used for performance evaluation in the static and wearable ECG monitoring conditions, respectively. The results show that our method outperforms other state-of-the-art models with an accuracy of 99.14% on the MITDB and maintains robustness with an accuracy of 98.68% in the wearable single-lead ECG analysis.

Clouds, cloud shadows (CCS), and numerous other factors will cause a missing data problem in passive remote sensing images. A well-known reconstruction method is the selection of a similar pixel (with an additional clear reference image) from the remaining clear part of an image to replace the missing pixel. Due to the merit of filling the missing value using a pixel acquired on the same image with the same sensor and the same date, this method is suitable for time-series applications when a time-series profile-based similar measure is utilized for selecting the similar pixel. Since the similar pixel is independently selected, the improper reference pixel or various accuracies obtained by different land covers causes the problem of salt-and-pepper noise in the reconstructed part of an image. To overcome these problems, this paper presents a spectral–temporal patch (STP)-based missing area reconstruction method for time-series images. First, the STP, the pixels of which have similar spectral and temporal evolution characteristics, is extracted using multi-temporal image segmentation. However, some STP have Missing Observations (STPMO) in the time series, which should be reconstructed. Next, for an STPMO, the most similar STP is selected as the reference STP; then, the mean and standard deviation of the STPMO is predicted using a linear regression method with the reference STP. Finally, the textural information, which is denoted by the spatial configuration of color or intensities of neighboring pixels, is extracted from the clear temporal-adjacent STP and “injected” into the missing area to obtain synthetic cloud-free images. We performed an STP-based missing area reconstruction experiment in Jiangzhou, Chongzuo, Guangxi with time-series images acquired by wide field view (WFV) onboard Chinese Gao Fen 1 on 12 different dates. The results indicate that the proposed method can effectively recover the missing information without salt-and-pepper noise in the reconstructed area; also, the reconstructed part of the image is consistent with the clear part without a false edge. The results confirm that the spectral information from the remaining clear part of the same image and textural information from the temporal-adjacent image can create seamless time-series images.