This review paper focuses on vision and touch-based sensors known as visuotactile. The study of visuotactile sensation and perception became a multidisciplinary field of study by philosophers, psychologists, biologists, engineers, technologists, and roboticists in the fields of haptics, machine vision, and artificial intelligence and it dates back centuries. To the best of our knowledge, the earliest records of visuotactile sensor was not applied to robotics and was not even for hand or finger imprint analysis yet for recording the foot pressure distribution of a walking or standing human known as pedobarograph. Our review paper presents the different literature related to visuotactile sensors that lead to a high-resolution miniature pedobarograph-like sensor known as the GelSight sensor. Moreover, this review paper focuses on architecture, different techniques, hardware, and software development of GelSight sensor since 2009 with its applications in haptics, robotics, and computer vision.

This study describes the creation of a new type of compact skin-like silicone-based thermosensitive visuotactile sensor based on GelSight technology. The easy integration of this novel sensor into a complex visuotactile system capable of very rapid detection of temperature change (30°C/s) is unique in providing a system that parallels the withdrawal reflex of the human autonomic system to extreme heat. To the best of authors’ awareness, this is the first time a sensor that can trigger a sensory impulse like a withdrawal reflex of humans in robotics community. To attain this, we used thermochromic pigments color blue, orange, and black with a threshold of 31°C, 43°C, and 50°C, respectively on the gel material. Each pigment has the property of becoming translucent when its temperature threshold is reached, making it possible to stack thermochromic pigments of different colors and thresholds. The pigments were air-brushed on a low-cost commercially available transparent silicone sponge. We used MobileNetV2 and transfer learning to simulate tactile preprocessing in order to recognize five different objects. The new thermosensitive visuotactile sensor helped to achieve 97.3% tactile image classification accuracy of five different objects. Our novel thermosensitive visuotactile sensor could be of benefit in material texture analysis, telerobotics, space exploration, and medical applications.

During open surgery, a surgeon relies not only on the detailed view of the organ being operated upon and on being able to feel the fine details of this organ but also heavily relies on the combination of these two senses. In laparoscopic surgery, haptic feedback provides surgeons information on interaction forces between instrument and tissue. There have been many studies to mimic the haptic feedback in laparoscopic-related telerobotics studies to date. However, cutaneous feedback is mostly restricted or limited in haptic feedback-based minimally invasive studies. We argue that fine-grained information is needed in laparoscopic surgeries to study the details of the instrument’s end and can convey via cutaneous feedback. We propose an exoskeleton haptic hand wearable which consists of five 4 × 4 miniaturized fingertip actuators, 80 in total, to convey cutaneous feedback. The wearable is described as modular, lightweight, Bluetooth, and WiFi-enabled, and has a maximum power consumption of 830 mW. Software is developed to demonstrate rapid tactile actuation of edges; this allows the user to feel the contours in cutaneous feedback. Moreover, to demonstrate the idea as an object displayed on a flat monitor, initial tests were carried out in 2D. In the second phase, the wearable exoskeleton glove is then further developed to feel 3D virtual objects by using a virtual reality (VR) headset demonstrated by a VR environment. Two-dimensional and 3D objects were tested by our novel untethered haptic hand wearable. Our results show that untethered humans understand actuation in cutaneous feedback just in a single tapping with 92.22% accuracy. Our wearable has an average latency of 46.5 ms, which is much less than the 600 ms tolerable delay acceptable by a surgeon in teleoperation. Therefore, we suggest our untethered hand wearable to enhance multimodal perception in minimally invasive surgeries to naturally feel the immediate environments of the instruments.

Robot-assisted guiding is gaining increased interest due to many applications involving moving in the noisy and low visibility environments. In such cases, haptic feedback is the most effective medium to communicate. In this letter, we focus on perturbation-based haptic feedback due to applications like guide dogs for visually impaired people and potential robotic counterparts providing haptic feedback via reins to assist indoor fire fighting. Since proprioceptive sensors like spindles and tendons are part of the muscles involved in the perturbation, haptic perception becomes a coupled phenomenon with spontaneous reflex muscle activity. The nature of this interplay and how the model-based sensory-motor integration evolves during haptic-based guiding is not well understood yet. We asked human followers to hold the handle of a hard rein attached to a one-DoF robotic arm that gave perturbations to the hand to correct an angle error of the follower. We found that followers start with a second-order reactive autoregressive following model and changes it to a predictive model with training. The reduction in cocontraction of muscles and leftward/rightward asymmetry of a set of followers behavioral metrics show that the model-based prediction accounts for the internal coupling between proprioception and muscle activity during perturbation responses.