In the field of multi-modal human-computer interaction (HCI), hand (and finger) motion tracking remains a critical challenge, because it is the input means of visual reproduction and provides the necessary parameters for real-time modeling of the haptic model. Among motion tracking methods including optical tracking methods, inertial tracking methods and magnetic motion tracking (MMT) methods, MMT method has a great development prospect as a marker-based method because of its advantages of no line-of-sight problem, being natural-oriented, and high accuracy. In this paper, we first discuss the performance and application scenarios under different configurations of electromagnet-based MMT and permanent-magnet-based MMT systems. And then, the MMT algorithms are reviewed, which can be divided into two broad categories: The model-based algorithms which include linear algorithm, nonlinear optimization algorithm, and recursive Bayesian algorithm; and the offline-data-based algorithms including neural network algorithm and look-up-table algorithm. In addition, a state-of-the-art comparison of MMT algorithms is given. In the end, we have an insightful discussion and give the expected future outlook of MMT.

Capturing finger joint angle information has important applications in human–computer interaction and hand function evaluation. In this paper, a novel wearable data glove is proposed for capturing finger joint angles. A sensing unit based on a grating strip and an optical detector is specially designed for finger joint angle measurement. To measure the angles of finger joints, 14 sensing units are arranged on the back of the glove. There is a sensing unit on the back of each of the middle phalange, proximal phalange, and metacarpal of each finger, except for the thumb. For the thumb, two sensing units are distributed on the back of the proximal phalange and metacarpal, respectively. Sensing unit response tests and calibration experiments are conducted to evaluate the feasibility of using the designed sensing unit for finger joint measurement. Experimental results of calibration show that the comprehensive precision of measuring the joint angle of a wooden finger model is 1.67%. Grasping tests and static digital gesture recognition experiments are conducted to evaluate the performance of the designed glove. We achieve a recognition accuracy of 99% by using the designed glove and a generalized regression neural network (GRNN). These preliminary experimental results indicate that the designed data glove is effective in capturing finger joint angles.

Tactile sensing is a particularly important and challenging task for a modern robot to safely manipulate objects, interact with humans in a shared space, and provide various services. This paper presents a 3D tactile glove for robots with the combination of a piezoresistive-based force sensor array (412 sensors) covering the full hand and a resistive bend sensor array (5 sensors) on the back of five fingers. Deep learning-based CNN (Convolutional Neural Network) methods using the designed tactile glove are proposed for object recognition. In the experiment for recognizing 15 objects with a dexterous robot hand, an average classification accuracy of 93.67% has been achieved. Comparison experiments with three other typical classifiers (the Quadratic SVM, Weighted KNN, and Bagged Trees) and our CNN methods show an average recognition accuracy of 91.67% with the 3D tactile glove, revealing an accuracy improvement of 4.17% over only using the force sensor array. We further apply our 3D tactile glove and the multi-modal CNN to identify three other objects and demonstrate their generalization ability of tactile object recognition with an average success accuracy of 78.33%. The proposed 3D tactile glove can be further used in human-robot interactions, the design of prosthetics and humanoid robots, and for improving the intelligence level in brain-computer collaborative systems.

Haptic rendering enables people to touch, perceive, and manipulate virtual objects in a virtual environment. Using six cascaded identical hollow disk electromagnets and a small permanent magnet attached to an operator's finger, this paper proposes and develops an untethered haptic interface through magnetic field control. The concentric hole inside the six cascaded electromagnets provides the workspace, where the 3D position of the permanent magnet is tracked with a Microsoft Kinect sensor. The driving currents of six cascaded electromagnets are calculated in real-time for generating the desired magnetic force. Offline data from an FEA (finite element analysis) based simulation, determines the relationship between the magnetic force, the driving currents, and the position of the permanent magnet. A set of experiments including the virtual object recognition experiment, the virtual surface identification experiment, and the user perception evaluation experiment were conducted to demonstrate the proposed system, where Microsoft HoloLens holographic glasses are used for visual rendering. The proposed magnetic haptic display leads to an untethered and non-contact interface for natural haptic rendering applications, which overcomes the constraints of mechanical linkages in tool-based traditional haptic devices.

This paper introduces a novel capacitive sensor array designed for tactile perception applications. Utilizing an all-in-one inkjet deposition printing process, the sensor array exhibited exceptional flexibility and accuracy. With a resolution of up to 32.7 dpi, the sensor array was capable of capturing the fine details of touch inputs, making it suitable for applications requiring high spatial resolution. The design incorporates two multiplexers to achieve a scanning rate of 100 Hz, ensuring the rapid and responsive data acquisition that is essential for real-time feedback in interactive applications, such as gesture recognition and haptic interfaces. To evaluate the performance of the capacitive sensor array, an experiment that involved handwritten number recognition was conducted. The results demonstrated that the sensor accurately captured fingertip inputs with a high precision. When combined with an Auxiliary Classifier Generative Adversarial Network (ACGAN) algorithm, the sensor system achieved a recognition accuracy of 98% for various handwritten numbers from “0” to “9”. These results show the potential of the capacitive sensor array for advanced human–computer interaction applications.

Because of the absence of visual perception, visually impaired individuals encounter various difficulties in their daily lives. This paper proposes a visual aid system designed specifically for visually impaired individuals, aiming to assist and guide them in grasping target objects within a tabletop environment. The system employs a visual perception module that incorporates a semantic visual SLAM algorithm, achieved through the fusion of ORB-SLAM2 and YOLO V5s, enabling the construction of a semantic map of the environment. In the human–machine cooperation module, a depth camera is integrated into a wearable device worn on the hand, while a vibration array feedback device conveys directional information of the target to visually impaired individuals for tactile interaction. To enhance the system’s versatility, a Dobot Magician manipulator is also employed to aid visually impaired individuals in grasping tasks. The performance of the semantic visual SLAM algorithm in terms of localization and semantic mapping was thoroughly tested. Additionally, several experiments were conducted to simulate visually impaired individuals’ interactions in grasping target objects, effectively verifying the feasibility and effectiveness of the proposed system. Overall, this system demonstrates its capability to assist and guide visually impaired individuals in perceiving and acquiring target objects.