The grasping of transparent objects is challenging but of significance to robots. In this article, a visual–tactile fusion framework for transparent object grasping in complex backgrounds is proposed, which synergizes the advantages of vision and touch, and greatly improves the grasping efficiency of transparent objects. First, we propose a multiscene synthetic grasping dataset named SimTrans12 K together with a Gaussian-mask annotation method. Next, based on the TaTa gripper, we propose a grasping network named transparent object-grasping convolutional neural network for grasping position detection, which shows good performance in both synthetic and real scenes. Inspired by human grasping, a tactile calibration method and a visual–tactile fusion classification method are designed, which improve the grasping success rate by 36.7% compared with direct grasping and the classification accuracy by 39.1%. Furthermore, the tactile height sensing module and the tactile position exploration module are added to solve the problem of grasping transparent objects in irregular and visually undetectable scenes. The experimental results demonstrate the validity of the framework.
15 December 2022 in IEEE Robotics and Automation Letters
Sequential multi-step cloth manipulation is a challenging problem in robotic manipulation, requiring a robot to perceive the cloth state and plan a sequence of chained actions leading to the desired state. Most previous works address this problem in a goal-conditioned way, and goal observation must be given for each specific task and cloth configuration, which is not practical and efficient. Thus, we present a novel multi-step cloth manipulation planning framework named Foldformer. Foldformer can complete similar tasks with only a general demonstration and utilize a space-time attention mechanism to capture the instruction information behind this demonstration. We experimentally evaluate Foldsformer on four representative sequential multi-step manipulation tasks and show that Foldsformer significantly outperforms state-of-the-art approaches in simulation. Foldformer can complete multi-step cloth manipulation tasks even when configurations of the cloth (e.g., size and pose) vary from configurations in the general demonstrations. Furthermore, our approach can be transferred from simulation to the real world without additional training or domain randomization. Despite training on rectangular clothes, we also show that our approach can generalize to unseen cloth shapes (T-shirts and shorts).
17 April 2023 in IEEE Robotics and Automation Letters
Transparent objects are a common part of daily life, but their unique optical properties make estimating their 6D pose a challenging task. In this letter, we propose TGF-Net, a monocular instance-level 6D pose estimation method for transparent objects based on geometric fusion. TGF-Net learns the edge features and surface fragments of transparent objects as intermediate features and reduces the influence of appearance changes on the 6D pose estimation of transparent objects by fusing rich geometric features in the network. Moreover, we propose an approach for generating high-fidelity large-scale synthetic datasets of transparent objects using Blender and use this approach to generate a synthetic dataset Trans6D-32 K. Trans6D-32 K contains rendered RGB images and poses information about transparent objects in a variety of different backgrounds, perspectives, and lighting conditions. To evaluate the performance of TGF-Net on 6D pose estimation of transparent objects, we compare with multiple related works on the dataset Trans6D-32 K. TGF-Net can be trained entirely on synthetic datasets without fine-tuning and applied directly to real-world scenarios. Multiple challenging real-scene experiments demonstrate the good performance of TGF-Net, while grasping experiments demonstrate the application value of TGF-Net in transparent object manipulation.
5 April 2023 in IEEE Robotics and Automation Letters
This paper focuses on the shape control manipulation of deformable linear objects (DLO) with a dual-arm robotic system. One significant challenge of DLO shape control is the underactuated control system, which means that finite robotic manipulators can not fully control DLO's shape due to the lack of sufficient constraints on DLO. We propose a novel DLO shape control framework aiming to stably control DLO's shape in a wide deformation range, which innovatively provides additional constraints on DLO via external contact. The proposed framework consists of a learning module named Learning Graph Dynamics with External Contact for deformable linear objects (LG-DEC) and an action generation module. The LG-DEC uses a graph convolution network to approximate the dynamics between DLO and contactable objects based on keypoint-based representation. In the action generation module, the learned graph dynamics is used to predict whether the randomly sampled actions can drive DLO closer to the desired shape and output the best action for DLO shape control. In simulation, the proposed framework accomplishes the DLO shape control task with external contact in six scenarios. Furthermore, the proposed framework can be well applied to a physical robotic system for the shape control on different types of DLO. Videos are available at https://sites.google.com/view/lg-dec .
29 February 2024 in IEEE Transactions on Multimedia
The acquisition of transparent 3D shapes will facilitate many multimedia and computer vision tasks, such as game/movie production and virtual enrioment applications. In this work, we propose a novel method for detailed reconstruction of transparent objects by exploiting polarimetric cues. Most of existing transparent shapes reconstruction methods usually lack sufficient constraints and suffer from the over-smooth problem. Hence, we introduce polarization information as a complementary cue. Specifically, we employ the implicit representation for object's geometry with a neural network, while the polarization render is capable of differentiably rendering the object's polarization images from given illumination configuration. However, direct comparison of rendered polarization images to the real-world captured images will have additional errors due to the transmission in the transparent object. To make the polarimetric cues technically feasible on transparent shapes reconstruction, the concept of reflection percentage which represents proportion of the reflection component is introduced as the weight of the polarization loss. Based on controllable environment setup, we build a polarization dataset containing several solid and smooth transparent objects to verify our method. Experimental results show that our method is capable of recovering detailed shapes and improving reconstruction quality of transparent objects.
1 June 2020 in IEEE Transactions on Instrumentation and Measurement
Vision-based measurement and prediction (VMP) are the very important and challenging part for autonomous robotic manipulation, especially in dynamic and uncertain scenarios. However, due to potential limitations of visual measurement in such an environment such as occlusion, lighting and hardware limitations, it is not easy to acquire the accurate positions of object as the observations. Moreover, manipulating a dynamic object with unknown or uncertain motion rules usually requires an accurate prediction of motion trajectory at the desired moment, which dramatically increases the difficulty. To address the problem, we propose a time-granularity based vision prediction framework whose core is an integrated prediction model based on multiple LSTM (i.e., Long Short-Term Memory) neural networks. At first, we use the vision sensor to acquire raw measurements and adopt the pre-processing method (e.g., data completion, error compensation and filtering) to turn raw measurements into the standard trajectory data. Then, we devise a novel integration strategy based on time-granularity boost (TGBoost) to select appropriate base predictors and further utilize these history trajectory data to construct the high-precision prediction model. Finally, we use the simulation and a series of dynamic manipulation experiments to validate the proposed methodology. The results also show our method outperforms state-of-the-art prediction algorithms in terms of prediction accuracy, success rate and robustness.