Transferring optical tactile skills learned from simulated environments to the real world benefits many robotic tactile applications, which can reduce the cost of data collection. However, the models purely trained on simulated data are often difficult to generalize well to the unseen real world due to the domain gap between the faultless training images and testing images with unpredictable manufacturing defects or natural wear. In this paper, we propose an Adaptively Correlation-attentive and Task-related Network (ACTNet) for tactile image transfer, a novel unsupervised adversarial domain adaptation method to narrow the domain gap for pixel-level tactile perception tasks. An adaptively correlative attention mechanism is introduced to improve the generator, which is capable of leveraging global information and focusing on salient regions. We also construct a task-related constraint loss based on the tactile image classification task to facilitate the zero-shot sim-to-real transfer of the tactile image classifier. Extensive experiment results on tactile image generation and tactile image classification indicate that our method outperforms the state-of-the-art method. Classification accuracy of 92.85% in the real-world environment is achieved without using any real-world classification labels.
1 November 2021 in Robotics and Autonomous Systems
Robust depth completion of transparent objects would be beneficial for industrial automation such as vision-based robotic grasping and manipulation. However, although some methods try to learn a compact intra-layer feature representation with the boost of the attention mechanism or the vision Transformer, they ignore the neglected corner regions and sparse geometry information that are important for accurate depth completion. To tackle these issues, we propose a novel sim-to-real transferable model, named CAGT, with interactive embedding aggregation and geometry awareness to reconstruct severely sparse depth maps of transparent objects in this paper. We design a Depth-clue Interaction Aggregation Module (DIAM) to enhance the Transformer’s ability to extract boundary corner features and thus supplement depth clues. Then, we propose a Geometric Information Augmentation Module (GIAM) to fuse the geometry-aware feature containing shape and surface details. Moreover, we introduce a contrastive learning mechanism to facilitate the sim-to-real generalization of the completion model. Extensive experiment results on two challenging datasets, ClearGrasp and TransCG, demonstrate that our proposed CAGT can obtain superior performance over the state-of-the-art methods. We also demonstrate that CAGT can improve the grasp accuracy of transparent objects by a robotic grasping generalization experiment. The code and supplementary video will be available at: https://github.com/xingshuojing/CAGT.
17 January 2023 in IEEE Transactions on Cognitive and Developmental Systems
Transferring robotic grasping skills learned from a simulator to the real world is beneficial in reducing the cost of labeling. However, the models trained on synthetic data are brittle when being applied to real-world data due to the domain gap. In this paper, we propose CCM Pixel-DA, a novel real-to-sim approach to unsupervised domain adaptation for object pose estimations, which outperforms conventional domain adaptation methods in preserving structural information, semantic information, and object pose during the transfer. The pipeline decouples domain adaptation and pose estimation, which allows the CCM Pixel-DA method to be integrated into state-of-the-art object pose estimation networks. The proposed method is further integrated into a pipeline for robot grasping. Experimental results on a real-world robot grasping system validate that the system is capable of grasping real-world objects without object pose annotations in the real-world domain.
14 June 2022 in IEEE Transactions on Cognitive and Developmental Systems
Learning manipulation skills from observing human demonstration video is a promising aspect for intelligent robotic systems. Recent advances in Video-to-Command provide an end-to-end approach to translate a video into robot plans. However, simultaneous Video-to-Command and action segmentation remain a major challenge for bimanual manipulations with fine-grained actions. Another concern is the generalization capability of end-to-end approaches in dealing with varied task parameters as well as environmental changes between the learned skills and the one-shot task demonstration for the robot to replay. In this paper, we propose a two-stream network for robots to learn and segment manipulation sub-actions from human demonstration videos. Our framework with the self-attention mechanism can segment learned skills and generate action commands simultaneously. To arrive at refined plans in situations of under-specified or redundant human demonstrations, we utilize PDDL-based skill scripts to model the semantics of demonstrated activities and infer latent movements. Experimental results on the extended manipulation dataset indicate that our approach generates more accurate commands than the state-of-the-art methods. Real-world experiment results on a Baxter robotic arm also demonstrated the feasibility of our method in reproducing fine-grained actions from video demonstrations.