The recognition prerformance of existing vision-based human action recognition (HAR) methods is greatly reduced in the case of low camera resolution or occlusion. Wearable sensors can provide complementary information to alleviate this problem. It is challenging to construct a robust HAR model using multimodal wearable-sensor data. In this paper, we propose a cross-Attention-based Multimodal fusion Wavelet Knowledge Distillation Network (MAWKDN) method to guide recognition from video data by acquiring complementary information from wearable sensors and reduce the noise effects through wavelet knowledge distillation, which improves the robustness of the model. A multi-attention dilated convolution kernel residual network including dilated convolution and an attention mechanism is constructed to extract features from various sensor modalities and fuse the various modal data through the cross-view attention method to acquire additional information from different modalities. To reduce the modal differences between different modalities of the teacher and student networks and acquire similar semantic knowledge, we learn the information between different modalities by constructing a graph structure of convolutional layer features, and computing the semantic preservation loss between the teacher and student networks. To reduce the influence of noise in the input data, we construct the loss of wavelet knowledge distillation, which transforms the image through the discrete wavelet transform and only retains the low frequency features to extract the useful information. The top-1 accuracy achieved on the UTD-MHAD (99.31%), Berkeley-MHAD (99.40%) and the F1-score on the MMAct (85.26% based on cross-session) dataset prove the superior performance of MAWKDN compared with the state-of-the-art HAR methods. Moreover, we demonstrate the robustness of the MAWKDN approach on the noise-added UTD-MHAD dataset.

Multimodal sensors, including vision sensors and wearable sensors, offer valuable complementary information for accurate recognition tasks. Nonetheless, the heterogeneity among sensor data from different modalities presents a formidable challenge in extracting robust multimodal information amidst noise. In this paper, we propose an innovative approach, named semantic-aware multimodal transformer fusion decoupled knowledge distillation method (SMTDKD), which not only guides video data recognition through the information interaction between different wearable-sensor data, but also through the information interaction between visual sensor data and wearable-sensor data, improving the robustness of the model. To preserve the temporal relationship within wearable-sensor data, the SMTDKD method converts them into 2D image data. Furthermore, a transformer-based multimodal fusion module is designed to capture diverse feature information from distinct wearable-sensor modalities. To mitigate modality discrepancies and encourage similar semantic features, graph cross-view attention maps are constructed across various convolutional layers to facilitate feature alignment. Additionally, semantic information is exchanged among the teacher-student network, the student network, and BERT-encoded labels. To obtain more comprehensive knowledge transfer, the decoupled knowledge distillation loss is utilized, thereby enhancing the generalization of the network. Experimental evaluations conducted on three multimodal datasets, namely UTD-MHAD, Berkeley-MHAD, and MMAct, demonstrate the superior performance of the proposed SMTDKD method over the state-of-the-art action human recognition methods.

In recent years, the field of action recognition using spatio-temporal graph convolution models for human skeletal data has made significant progress. However, current methodologies tend to prioritize spatial graph convolution, which leads to an underutilization of valuable information present in skeletal data. It limits the model’s ability to effectively capture complex data patterns, especially in time series data, ultimately impacting recognition accuracy significantly. To address the above issues, this paper introduces an Attention-based Semantic-guided Multi-stream Graph Convolution Network (ASMGCN), which can extract the deep features in skeletal data more fully. Specifically, ASMGCN incorporates a novel temporal convolutional module featuring an attention mechanism and a multiscale residual network, which can dynamically adjust the weights between skeleton graphs at different time points, enabling better capture of relational features. In addition, semantic information is introduced into the loss function, enhancing the model’s ability to distinguish similar actions. Furthermore, the coordinate information of different joints within the same frame is explored to generate new relative position features known as centripetal and centrifugal streams based on the center of gravity. These features are integrated with the original position and motion features of skeleton, including joints and bones, enriching the inputs to the GCN network. Experimental results on the NW-UCLA, NTU RGB+D (NTU60) and NTU RGB+D 120 (NTU120) datasets demonstrate that ASMGCN outperforms other state-of-the-art (SOTA) Human Action Recognition (HAR) methods, signifying its potential in advancing the field of action recognition using skeletal data.

Sequential recommendation has attracted a lot of attention from both academia and industry, however the privacy risks associated with gathering and transferring users’ personal interaction data are often underestimated or ignored. Existing privacy-preserving studies are mainly applied to traditional collaborative filtering or matrix factorization rather than sequential recommendation. Moreover, these studies are mostly based on differential privacy or federated learning, which often lead to significant performance degradation, or have high requirements for communication. In this work, we address privacy-preserving from a different perspective. Unlike existing research, we capture collaborative signals of neighbor interaction sequences and directly inject indistinguishable items into the target sequence before the recommendation process begins, thereby increasing the perplexity of the target sequence. Even if the target interaction sequence is obtained by attackers, it is difficult to discern which ones are the actual user interaction records. To achieve this goal, we introduce a novel sequential recommender system called CoLlaborative-cOnfusion seqUential recommenDer (CLOUD) , which incorporates a collaborative confusion mechanism to modify the raw interaction sequences before conducting recommendation. Specifically, CLOUD first calculates the similarity between the target interaction sequence and other neighbor sequences to find similar sequences. Then, CLOUD considers the shared representation of the target sequence and similar sequences to determine the operation to be performed: keep, delete, or insert. A copy mechanism is designed to make items from similar sequences have a higher probability to be inserted into the target sequence. Finally, the modified sequence is used to train the recommender and predict the next item. We conduct extensive experiments on three benchmark datasets. The experimental results show that CLOUD achieves a maximum modification rate of 66.57% on interaction sequences and obtains over 99% recommendation accuracy compared to the state-of-the-art sequential recommendation methods. This proves that CLOUD can effectively protect user privacy at minimal recommendation performance cost, which provides a new solution for privacy-preserving for sequential recommendation. Our implementation is available at https://github.com/weiwang0927/CLOUD .

The widespread deployment of dual-camera systems has laid a solid foundation for practical applications of infrared (IR)-RGB cross-modality person reidentification (ReID). However, the inherent modality differences between RGB and IR images cause significant intra-class variances in the feature space for individuals of the same identity. Current methods typically employ various network architectures for the image style transfer or extracting modality-invariant features, yet they overlook the information extraction from the most fundamental spectral semantic features. Based on the existing approaches, we propose a multi-spectral semantic alignment (MSSA) architecture aimed at aligning fine-grained spectral semantic features across both intra-modality and inter-modality perspectives. Through modality center semantic alignment (MCSA) learning, we comprehensively mitigate differences in identity features of different modalities. Moreover, to attenuate the discriminative information unique to a single modality, we introduce the modality reliability intensification (MRI) loss to enhance the reliability of identity information. Finally, to tackle the challenge that inter-modality intra-class disparities surpass inter-modality inter-class differences, we leverage the dynamic discriminative center (DDC) loss to further bolster the discriminability of reliable information. Through an extensive experiments conducted on SYSU-MM01, RegDB, and LLCM datasets, we demonstrate the substantial advantages of the proposed MSSA over other state-of-the-art methods.