Visuotactile sensors have significant advantages in high-resolution contact shape sensing, which is crucial for robots to locate and manipulate objects. However, high precision and robust 3-D geometry sensing is still a challenging problem. In this article, we propose an improved GelStereo sensor, named GelStereo 2.0, with a more stable and compact structure. A refractive stereo ray tracing modeling method is proposed to eliminate 3-D geometry reconstruction errors caused by multimedium light refraction. In addition, we introduce a novel calibration method named multiple contact depths to obtain refractive parameters, which only needs a few pressings using calibration boards. Extensive experimental results show that the measurement error of contact surface depth is less than 0.15 mm. Furthermore, two well-calibrated sensors mounted on the gripper are successfully applied to a fingertip bolt screwing task, which demonstrates the effectiveness of the GelStereo 2.0 sensor and the proposed tactile 3-D geometry sensing method.

Vision-based tactile sensors have recently shown promising contact information sensing capabilities in various fields, especially for dexterous robotic manipulation. However, dense contact geometry measurement is still a challenging problem. In this article, we update the design of our previous GelStereo tactile sensor and present a self-supervised contact geometry learning pipeline. Specifically, a self-supervised stereo-based depth estimation neural network (GS-DepthNet) is proposed to achieve real-time disparity estimation, and two specifically designed loss functions are proposed to accelerate the convergence of the network during the training process and improve the inference accuracy. Furthermore, extensive qualitative and quantitative experiments of perceived contact shape were performed on our GelStereo sensor. The experimental results verify the accuracy and robustness of the proposed contact geometry sensing pipeline. This updated GelStereo tactile sensor with dense contact geometric sensing capability has predictable application potential in the field of industrial and service robots.

Recently, visuotactile sensors have shown promising potential in robotics due to their high-resolution sensing ability. Unfortunately, the majority of available visuotactile sensors are limited to flat shapes, which severely limits their application possibilities. In this article, we propose a novel curved visuotactile sensor, the GelStereo Palm, which senses the 3-D contact geometry on a curved surface using a binocular vision system. Meanwhile, to solve the light refraction problem in the binocular stereo vision system under a curved medium, a refractive stereo ray tracing model for GelStereo Palm is presented. Moreover, a 3-D tactile point cloud sensing pipeline is introduced to reconstruct the 3-D contact geometry in real-time. Finally, extensive experiments are conducted to verify the accuracy and robustness of the 3-D contact geometry sensing of our GelStereo Palm sensor.

Endowing the robot with tactile perception can effectively improve manipulation dexterity, along with various benefits of human-like touch. Using GelStereo (GS) tactile sensing, which gives high-resolution contact geometry information, including 2-D displacement field, and 3-D point cloud of the contact surface, we present a learning-based slip detection system in this study. The results reveal that the well-trained network achieves 95.79% accuracy on the never-seen testing dataset, which surpasses the current model-based and learning-based methods using visuotactile sensing. We also propose a general framework for slip feedback adaptive control for dexterous robot manipulation tasks. The experimental results show the effectiveness and efficiency of the proposed control framework using GS tactile feedback when deployed on real-world grasping and screwing manipulation tasks on various robot setups.

High-resolution visuotactile sensors are often bulky and have flat contact shapes, making it difficult to integrate into dexterous multifinger fingertips and perform manipulation tasks smoothly. In this article, we present a soft, low-cost, biomimetic fingertip-sized visuotactile sensor, which can perceive high-precision 3-D surface deformation. This novel sensor, named GelStereo BioTip, adopts stereo vision system to robustly acquire 3-D deformation information on multicurvature contact surfaces. In particular, a self-calibrating refractive stereo ray tracing modeling method is proposed to solve the problem of ray refraction on multimedium and multicurvature refractive surfaces in the binocular reconstruction system. GelStereo BioTip has an overall spatial resolution of 1.2 mm, a shape measurement accuracy of less than 0.28 mm over a range of 0–5 mm for numerous distinct contacts with varying contact area. We integrate the bionic fingertips into a dexterous robotic hand for validation, and the proposed bionic fingertip concept can be seamlessly transferred to a variety of robotic manipulators.

GelStereo sensing technology is capable of performing three-dimensional (3D) contact shape measurement under various contact structures such as bionic curved surfaces, which has promising advantages in the field of visuotactile sensing. However, due to multi-medium ray refraction in the imaging system, robust and high-precision tactile 3D reconstruction remains a challenging problem for GelStereo-type sensors with different structures. In this paper, we first propose a universal Refractive Stereo Ray Tracing (RSRT) model for GelStereo-type sensing systems to realize 3D reconstruction of the contact surface. Moreover, a relative geometry-based optimization method is presented to calibrate multiple parameters of the proposed RSRT model, such as the refractive indices and structural dimensions. Furthermore, extensive quantitative calibration experiments are performed on four different GelStereo sensing platforms; the experimental results show that the proposed calibration pipeline can achieve less than 0.35 mm in Euclidean distance error, based on which we believe that the proposed refractive calibration method can be further applied in more complex GelStereo-type and other similar visuotactile sensing systems. Such high-precision visuotactile sensors can facilitate the study of robotic dexterous manipulation.