Deep reinforcement learning (DRL) has achieved great success across multiple fields; however, in the field of robot control, the acquisition of large amounts of motion data from real robots is challenging. In this work, an algorithm is proposed to train a neural network model with a large amount of data in a simulated environment and then transfer the model to the real environment. Proximal policy optimization (PPO) is used to train the agent in simulator, and generative adversarial imitation learning (GAIL) is specified to transfer the model to the real world. The algorithm can guide the two-armed robot to complete the task well in the face of complex assembly tasks. A total of three tasks of different difficulties were set to test the performance of the algorithm. In a large experimental study, the proposed algorithm outperforms other algorithms, and the real robot arm completes the assembly task significantly faster than script and keyboard operations.

In modern manufacturing, making accurate and timely decisions requires the ability to effectively handle multiple types of data. This paper presents a multimodal system designed specifically for smart manufacturing applications. The system combines various data sources including images, sensor data, and production records, using advanced multimodal large language models. This approach addresses common limitations of traditional single-modal methods, such as isolated data analysis and poor integration between different data types. Key contributions include a unified method for representing different data types, dynamic semantic tokenization for better data processing, strong alignment strategies across modalities, and a practical two-stage training method involving initial large-scale pretraining and later fine-tuning for specific tasks. Additionally, a novel Transformer-based model is introduced for generating both images and text, significantly improving real-time decision-making capabilities. Experiments on relevant industrial datasets show that this method consistently performs better than current state-of-the-art approaches in tasks like image–text retrieval and visual question answering. The results demonstrate the effectiveness and versatility of the proposed methods, offering important insights and practical solutions to enhance intelligent manufacturing, predictive maintenance, and anomaly detection, thus supporting the development of more efficient and reliable industrial systems.

Objective. Recent technological advances show the feasibility of fusing surface electromyography (sEMG) signals and movement data to predict lower limb ambulation intentions. However, since the invasive fusion of different signals is a major impediment to improving predictive performance, searching for a non-invasive (NI) fusion mechanism for lower limb ambulation pattern recognition based on different modal features is crucial. Approach . We propose an end-to-end sequence prediction model with NI dual attention temporal convolutional networks (NIDA-TCNs) as a core to elegantly address the essential deficiencies of traditional decision models with heterogeneous signal fusion. Notably, the NIDA-TCN is a weighted fusion of sEMG and inertial measurement units with time-dependent effective hidden information in the temporal and channel dimensions using TCN and self-attentive mechanisms. The new model can better discriminate between walking, jumping, downstairs, and upstairs four lower limb activities of daily living. Main results . The results of this study show that the NIDA-TCN models produce predictions that significantly outperform both frame-wise and TCN models in terms of accuracy, sensitivity, precision, F1 score, and stability. Particularly, the NIDA-TCN with sequence decision fusion (NIDA-TCN-SDF) models, have maximum accuracy and stability increments of 3.37% and 4.95% relative to the frame-wise model, respectively, without manual feature-encoding and complex model parameters. Significance . It is concluded that the results demonstrate the validity and feasibility of the NIDA-TCN-SDF models to ensure the prediction of daily lower limb ambulation activities, paving the way to the development of fused heterogeneous signal decoding with better prediction performance.

Gait is a behavioral biometric trait that enables non-invasive person recognition based on individual walking patterns. Camera-based gait acquisition is convenient, but silhouette sequences often contain substantial motion-irrelevant appearance information, such as body shape, clothing, and carried objects. To address this problem, a multi-scale time series differencer is proposed to acquire tensor difference data between adjacent frames, so as to extract dynamic feature information in motion gait image sequences. Experiments on the CASIA-B dataset show that the proposed method achieves Rank-1 accuracies of 97.7%, 94.6%, and 80.0% under NM, BG, and CL conditions, respectively. Ablation results further demonstrate that MTDU improves the mean accuracy from 84.7% to 90.8% compared with single-scale temporal differencing. The multi-scale time series differencer shows potential for fields including sports motion gait detection and recognition, surveillance security motion gait identity authentication, and medical motion gait recovery assessment for sports injuries, demonstrating practical application value.