Motions of the fingers are complex since hand grasping and manipulation are conducted by spatial and temporal coordination of forearm muscles and tendons. The dominant methods based on surface electromyography (sEMG) could not offer satisfactory solutions for finger motion classification due to its inherent nature of measuring the electrical activity of motor units at the skin's surface. In order to recognize morphological changes of forearm muscles for accurate hand motion prediction, ultrasound imaging is employed to investigate the feasibility of detecting mechanical deformation of deep muscle compartments in potential clinical applications. In this study, finger motion classification has been represented as subproblems: recognizing the discrete finger motions and predicting the continuous finger angles. Predefined 14 finger motions are presented in both sEMG signals and ultrasound images and captured simultaneously. Linear discriminant analysis classifier shows the ultrasound has better average accuracy (95.88%) than the sEMG (90.14%). On the other hand, the study of predicting the metacarpophalangeal (MCP) joint angle of each finger in nonperiod movements also confirms that classification method based on ultrasound achieves better results (average correlation 0.89 ± 0.07 and NRMSE 0.15 ± 0.05) than sEMG (0.81 ± 0.09 and 0.19 ± 0.05). The research outcomes evidently demonstrate that the ultrasound can be a feasible solution for muscle-driven machine interface, such as accurate finger motion control of prostheses and wearable robotic devices.
9 December 2019 in IEEE Transactions on Industrial Informatics
Incidence of children with autism spectrum disorder (ASD) has increased with an average rate of 1% worldwide. Clinical ASD screening, especially for children screening is a laborious and skilled task, there is however no objective and effective method automating ASD children screening. Analysing children ASD characteristics in predefined motion behaviour protocols is attempted to provide automatic solutions to children ASD screening. A novel protocol, Response to Name (RTN), is proposed in this paper for ASD clinical validation and diagnosis. The RTN method is jointly designed with clinical partners, novel gaze estimation is developed for validating ASD characteristic behaviour. Seventeen subjects including 10 adults and 7 children (5 ASD subjects and 2 healthy subjects) participated the experiment. The experiment results showed that the proposed RTN system achieves an average classification score of 92.7% fully demonstrating that the principle of motion protocol based ASD screening has the potential to have early ASD screening automated.
2 September 2025 in IEEE Transactions on Instrumentation and Measurement
Since vision-based tactile sensors (VBTSs) are firstly proposed and developed, they have attracted more and more attentions in recent years. This survey aims to provide a comprehensive review of VBTSs, with emphasis on their algorithms and applications. First, I briefly introduce the concept, principle, and prototype of VBTSs. Then, I illustrate the deployment of VBTSs into robotic systems and discuss the state-of-the-art progress of VBTSs-based robotic grasping and manipulation, then outline the main functions of VBTSs and provide a systematic survey on their algorithms, with a focus on deep learning and computer vision. Furthermore, I summarize the applications of VBTSs in practical scenarios. Finally, I discuss key research challenges and envision future directions for VBTSs.
25 December 2024 in IEEE Transactions on Medical Robotics and Bionics
Since ultrasound signal is firstly used to build a human machine interface (HMI) for prosthetic control in 2006, ultrasound-based HMIs have received the great attention in the past 18 years. In this paper, I provide a comprehensive overview of every aspect of an ultrasound-based HMI for hand gesture recognition (HGR). Firstly, I introduce the principle of ultrasound-based HGR and then outline a workflow for an ultrasound-based HMI for HGR and detail each step involved in the workflow, followed by an introduction of performance evaluation and robustness of this type of HMI. Then, I review the research progress of ultrasound-based HMIs for HGR. After that, I introduce the state-of-the-art wearable ultrasound systems for HMIs. Furthermore, I summarize the miscellaneous application of ultrasound-based HMIs for HGR. Finally, I discuss the main research challenges and further envision future research directions for ultrasound-based HMIs for HGR.
Objective: Measurements of human body composition such as fat mass (FM) and fat-free mass (FFM) are critical for studying malnutrition and the effects of nutritional interventions. This study introduces research toward a novel ultrasound scanning protocol combined with a deep learning analysis pipeline for predicting body composition. Methods: We analyzed a clinical dataset of 65 premature infants, consisting of ultrasound images from three anatomical locations (biceps, abdomen, and quadriceps), and ground truth FM and FFM from air displacement plethysmography (ADP). Our investigation focused on determining: (1) the optimal data processing methods for this application; (2) suitable baseline deep learning models for prediction to guide our learning strategy; and (3) the anatomical locations and image regions most predictive of FM and FFM. Results: We demonstrate that: (1) pre-processing techniques such as denoising, median filtering, and data augmentation enhance performance; (2) by employing a modified EfficientNet-B1 architecture, we achieve fully automatic body composition predictions from ultrasound images; (3) images obtained from combinations of biceps and quadriceps, as well as biceps, quadriceps, and abdomen scanning locations, resulted in mean absolute percent error (MAPE) values of 26.1% and 25.32%, respectively. Finally, sensitivity analysis shows that FM and FFM prediction are influenced by different body parts, as well as adipose and muscle tissue thickness. Conclusion: This study represents the first demonstration of deep learning for automated human body composition prediction from ultrasound images and lays a critical foundation for a novel ultrasound scanning and interpretation protocol to assess malnutrition.
Ultrasound imaging (US imaging) is increasingly used to build a human machine interface (HMI). However, few studies investigate online performance of such HMI and its robustness against fatigue. This study evaluates the feasibility of using B-mode ultrasound as a real-time HMI and explore effects of muscle fatigue and force level variation on the classification performance of the ultrasound-based HMI. To achieve this, an online finger motion experiment and a muscle fatigue experiment were conducted. The real-time performance was evaluated with motion selection time (ST), completion time (CT), completion rate (CR), and real-time accuracy (RA). Results demonstrate that the achieved ST and CT meet requirements for online prosthetic control. Then, after an algorithm incorporating bag-of-features (BoF) model and speeded-up robust features (SURF) was introduced as well as gray-level co-occurrence matrix (GLCM), robustness against muscle fatigue and force level variation was evaluated with features derived from edge points based keypoints (BoF-EP) and dense grid based keypoints (BoF-DG). Results reveal that GLCM and BoF-EP achieve high classification accuracy (CA), indicating that they have a good robustness against muscle fatigue and force level variation. Additionally, muscle fatigue has a minor influence on the HMI’s classification performance, whereas force level variation affects it significantly.