In-hand object localization and manipulation has always been a challenging task in robotic community. In this article, we address this problem by vision-based tactile sensing with high-spatial resolution. Specifically, we design a novel tactile sensor based on stereo vision, named GelStereo, which can perceive tactile point cloud with high-spatial resolution ( $<\!1$ mm). A tactile-based in-hand object localization pipeline composed of saliency detection and probabilistic point-set registration algorithms of the perceived contact point cloud is presented. Furthermore, extensive qualitative and quantitative analyses of perceived tactile point cloud and in-hand localization and insertion experiments of small parts are performed on our robot platform. The experimental results verify the accuracy and robustness of the tactile point cloud sensed by the novel GelStereo tactile sensor and the proposed in-hand object localization pipeline. This novel high-resolution visuotactile sensing technology has predictable application potential in the field of dexterous robotic manipulation.

Predicting whether a particular grasp will succeed is critical to performing stable grasping and manipulating tasks. Robots need to combine vision and touch as humans do to accomplish this prediction. The primary problem to be solved in this process is how to learn effective visual-tactile fusion features. In this paper, we propose a novel Visual-Tactile Fusion learning method based on Self-Attention mechanism (VTFSA) to address this problem. We compare the proposed method with the traditional methods on two public multimodal grasping datasets, and the experimental results show that the VTFSA model achieves outperforming traditional methods by a margin of 5+% and 7+%. Furthermore, visualization analysis indicate that the VTFSA model can further capture some position-related visual-tactile fusion features that are beneficial to this task and is more robust compared to traditional methods.

Significant progress has been made in vision-language models. However, language-conditioned robotic manipulation for contact-rich tasks remains underexplored, particularly with respect to tactile s...

Vision-based tactile sensors have recently shown promising contact information sensing capabilities in various fields, especially for dexterous robotic manipulation. However, dense contact geometry measurement is still a challenging problem. In this article, we update the design of our previous GelStereo tactile sensor and present a self-supervised contact geometry learning pipeline. Specifically, a self-supervised stereo-based depth estimation neural network (GS-DepthNet) is proposed to achieve real-time disparity estimation, and two specifically designed loss functions are proposed to accelerate the convergence of the network during the training process and improve the inference accuracy. Furthermore, extensive qualitative and quantitative experiments of perceived contact shape were performed on our GelStereo sensor. The experimental results verify the accuracy and robustness of the proposed contact geometry sensing pipeline. This updated GelStereo tactile sensor with dense contact geometric sensing capability has predictable application potential in the field of industrial and service robots.

Visuotactile sensors have significant advantages in high-resolution contact shape sensing, which is crucial for robots to locate and manipulate objects. However, high precision and robust 3-D geometry sensing is still a challenging problem. In this article, we propose an improved GelStereo sensor, named GelStereo 2.0, with a more stable and compact structure. A refractive stereo ray tracing modeling method is proposed to eliminate 3-D geometry reconstruction errors caused by multimedium light refraction. In addition, we introduce a novel calibration method named multiple contact depths to obtain refractive parameters, which only needs a few pressings using calibration boards. Extensive experimental results show that the measurement error of contact surface depth is less than 0.15 mm. Furthermore, two well-calibrated sensors mounted on the gripper are successfully applied to a fingertip bolt screwing task, which demonstrates the effectiveness of the GelStereo 2.0 sensor and the proposed tactile 3-D geometry sensing method.

in{abstract} This study addresses the ability of an underwater vehicle-manipulator system (UVMS) to perceive submarine creatures in complex undersea environments by a vision system. A real-time system for Submarine Creature Perception and Positioning (SCPP) based on binocular vision system is proposed. In particular, we adopt a new Four-Corner Neighborhood Matching Method (FCNMM) to obtain specific target positions. A Fusion Correction Mechanism (FCM) is added to the Kernelized correlation filter (KCF) tracking algorithm to improve tracking performance. Furthermore, SCPP is applied in our UVMS platform and conducted seafloor creature harvesting at the Zhangzidao seafood farm in Dalian, China. The experimental results show the feasibility and robustness of the SCPP system.