Human machine interaction (HMI) is an interactive way of information exchange between human and machine. By collecting the information that can be conveyed by the person to express the intention, and then transforming and processing the information, the machine can work according to the intention of the person. However, the traditional HMI including mouse, keyboard etc. usually requires a fixed operating space, which limits people's actions and cannot directly reflect people's intentions. It requires people to learn systematically how to operate skillfully, which indirectly affects work efficiency. Hand gesture, as one of the important ways for human to convey information and express intuitive intention, has the advantages of high degree of differentiation, strong flexibility and high efficiency of information transmission, which makes hand gesture recognition (HGR) as one of the research hotspots in the field of HMI. In order to enable readers to systematically and quickly understand the research status of HGR and grasp the basic problems and development direction of HGR, this article takes the sensing method used by HGR technology as the entry point, and makes a detailed elaboration and systematic summary by referring to a large number of research achievements in recent years.

The development of robotic interaction is becoming an increasing topic. The excellent grasp ability of low safety and fragile objects will greatly solve the complex labour of people. Rigid actuation have made significant progress for manufacturing and automation assembly, but it lacks of excellent interaction ability for humans and unstructured environments. Soft manipulator is the basic part of man-machine safety cooperation, which can make up for the deficiency. This paper describes the current research status of soft manipulator, driving methods, structures of soft manipulator and principles. Then, we specially show the joint fluid-elastic pneumatic drive, which enables control bending of different joints of the fingers. A detailed classification and principle description of intelligent materials is made for the flexible hands application. In addition, the principle of two new driving modes: biological driving and gas-liquid driving are presented. The performance of different driving modes is compared in detail, and the advantages and disadvantages of different pneumatic driving modes are listed. Finally, we make some prediction for the application prospect of soft manipulator in the future. The practical application and key problems of soft manipulator are summarised endly.

Human-machine interface(HMI) technology has gradually become a research hotspot with the continuous development of computer technology and Internet of Things (IOT) technology. Hand gesture recognition (HGR) as an important part of HMI technology has been widely concerned. Among the many technical routes of HGR technology, the wearable HGR technology based on A-mode ultrasound shows great application potential due to its advantages such as lightweight device, free from sensors’ constraint etc. However, in the process of processing A-mode ultrasonic signal, the amplitude of the signal at the main position may vary due to muscle fatigue and strength etc. when the same gesture is repeated some time later. In this paper, we design a method to overcome this problem and accomplish HGR in offline state by setting the threshold and use normalization method for the energy feature of ultrasonic signal according to the threshold. Three machine learning algorithms, including support vector machine (SVM), linear discriminant analysis (LDA) and Naive Bayes (NB) were used to verify the feasibility of this method, and the average recognition accuracy in the experiment reached 95.26%. Signal stability is also verified to be improved. In addition, We design a recognition strategy based on NB algorithm in this paper so that the model can identify the unknown gestures (i.e. gestures that have not been trained by the model). Five known gestures and five unknown gestures were set in the experiment, the average recognition accuracy of the known gestures is 91.2%, and the unknown gestures is 81.8%. This method can be used to optimize the user experience of HGR system.

A-mode ultrasound has the advantages of high resolution, easy calculation and low cost in predicting dexterous gestures. In order to accelerate the popularization of A-mode ultrasound gesture recognition technology, we designed a human-machine interface that can interact with the user in real-time. Data processing includes Gaussian filtering, feature extraction and PCA dimensionality reduction. The NB, LDA and SVM algorithms were selected to train machine learning models. The whole process was written in C++ to classify gestures in real-time. This paper conducts offline and real-time experiments based on HMI-A (Human-machine interface based on A-mode ultrasound), including ten subjects and ten common gestures. To demonstrate the effectiveness of HMI-A and avoid accidental interference, the offline experiment collected ten rounds of gestures for each subject for ten-fold cross-validation. The results show that the offline recognition accuracy is 96.92% ± 1.92%. The real-time experiment was evaluated by four online performance metrics: action selection time, action completion time, action completion rate and real-time recognition accuracy. The results show that the action completion rate is 96.0% ± 3.6%, and the real-time recognition accuracy is 83.8% ± 6.9%. This study verifies the great potential of wearable A-mode ultrasound technology, and provides a wider range of application scenarios for gesture recognition.

At present, the research on gesture recognition based on ultrasonic signals is mostly about static gestures. Compared with static gestures, dynamic gestures reflect human intent more intuitively. However, the speed difference of dynamic gestures will cause a mismatch in time series. To tackle this problem, we proposed a dynamic gesture recognition scheme based on the dynamic time warping (DTW) algorithm. First, the ultrasonic signals of the forearm muscles were acquired by four A-mode ultrasonic transducers. Second, the original signals underwent Gaussian filtering, feature extraction, normalization, and principal component analysis (PCA) dimensionality reduction to obtain the main feature information and convert it into time series. Finally, the time series of dynamic gestures were matched based on the DTW algorithm, and the minimum Euclidean distance between different time series was calculated for prediction. We tested five dynamic gestures on ten subjects. The experiments were conducted for a total of ten rounds, and only one round was used as the training set. Results reveal that using the max feature can get an accuracy of 88.0% ± 5.9%. In addition, the energy feature (87.5% ± 6.4%) and the mean feature (85.2% ± 6.9%) also performed well. This research realizes the dynamic gesture recognition of A-mode ultrasonic for the first time, proving the potential of ultrasonic signals, and providing a wider range of use scenarios.

The gesture recognition technology as one of the human-machine interfaces can convenient and effectively express the intention of human, and has become the hot research hotspot in recent years. Force level is a key factor while gesture recognition for more dexterous and natural prosthetic control. To provide researchers with a systematic and quick overview of the relevant and future developments in gesture recognition and force assessment techniques, this review synthesizes current commonly used sensor interfaces, data processing methods, and methods that have improved recognition performance. The experimental design and related results of gesture recognition and force assessment with various types of sensors are analyzed and compared to understand the scope of application and recognition performance of different sensors. This review summarizes the challenges and future work in the five areas of hardware, use environment, broad applicability, physiological factors, and comfort of use in practical applications. Finally, the conclusion prospects that future research may need to focus on improving model generalization and robustness to environmental, physiological factors, etc. by building large datasets and developing flexible, long-lasting, lightweight and senseless, high-performance interfaces.