18 April 2023 in IEEE Sensors Journal
The variation in the distributions of recorded data between individuals leads to low classification accuracy. To address this issue, we introduce a multimodal fusion convolutional neural network (MFCNN). This network extracts common information from surface electromyography (sEMG) and accelerometer signals of different subjects using a two-stream CNN. To enhance the classification accuracy of a particular subject, a fine-tuning approach was implemented. The performance of the proposed method was assessed in four different scenarios, which include inter-subject classification, inter-subject classification when training data from multiple subjects, fine-tuned inter-subject classification, and fine-tuned inter-subject classification when training data from multiple subjects. The results demonstrate that in the inter-subject scenario, when multiple subjects are available for training, the MFCNN achieves higher classification accuracy (p < 0.05) than other neural networks and support vector machines (SVMs) that use sEMG signals (NN and SVM), accelerometer signals (accNN and accSVM), sEMG and accelerometer signals (MFNN and MFSVM) as inputs, as well as a CNN that uses sEMG signals as input after fine-tuning. Furthermore, compared with an MFCNN model trained with data from a single subject and an accCNN model trained with data from a single subject or multiple subjects, an MFCNN trained with multiple subjects demonstrated better performance on new subjects after fine-tuning (p < 0.05). This method can learn common features among different subjects and improve the performance of classification among subjects. Our proposed method demonstrates the innovation of using a multimodal fusion approach and two-stream CNN to improve inter-subject classification accuracy in upper limb movements.
27 March 2023 in IEEE Transactions on Neural Systems and Rehabilitation Engineering
Convolutional neural network (CNN)-based models are widely used in human movement decoding based on surface electromyography. However, they capture only the spatial information of the surface electromyography and lack prior knowledge of the system, resulting in unsatisfactory decoding accuracy. To address these issues, we propose an attention-based Kalman filter scheme (AKFS), which uses an attention-based CNN model to better extract temporal information and a KF to add prior knowledge of the system. We further solve the problem of insufficient data due to the short training time of new subjects by using a transfer learning method based on a fine-tuning strategy. The proposed scheme was tested in four scenarios: intra-session, intra-session long-time use, inter-subject, and inter-subject with a fine-tuning strategy. The proposed attention-based CNN model outperformed the vanilla CNN model and a hybrid CNN-long short-term memory (LSTM) model in intra-session and intra-session long-time use. After fine-tuning with a small amount of data on a new subject, the attention-based CNN model achieved higher decoding accuracy than the vanilla CNN model and lower response time than CNN-LSTM model. Furthermore, the schemes with KF outperformed the schemes without KF in all scenarios. Our proposed scheme improves the decoding accuracy of the traditional CNN model for a single subject by better capturing the temporal information of the surface electromyography signal and increasing the prior knowledge of the system. Additionally, the proposed scheme can be easily transferred to a new subject using only a small amount of data.