6 November 2020 in IEEE Robotics and Automation Letters
We present a target-driven navigation system to improve mapless visual navigation in indoor scenes. Our method takes a multi-view observation of a robot and a target image as inputs at each time step to provide a sequence of actions that move the robot to the target without relying on odometry or GPS at runtime. The system is learned by optimizing a combinational objective encompassing three key designs. First, we propose that an agent conceives the next observation before making an action decision. This is achieved by learning a variational generative module from expert demonstrations. We then propose predicting static collision in advance, as an auxiliary task to improve safety during navigation. Moreover, to alleviate the training data imbalance problem of termination action prediction, we also introduce a target checking module to differentiate from augmenting navigation policy with a termination action. The three proposed designs all contribute to the improved training data efficiency, static collision avoidance, and navigation generalization performance, resulting in a novel target-driven mapless navigation system. Through experiments on a TurtleBot, we provide evidence that our model can be integrated into a robotic system and navigate in the real world. Videos and models can be found in the supplementary material.
16 August 2022 in IEEE Transactions on Instrumentation and Measurement
In the field of aircraft maintenance, regular inspection of fuselage surface during the aircraft life cycle is a vital task to ensure the aircraft quality and flight safety. Currently, the inspection task is generally carried out manually in an indoor hangar, which is with low efficiency and reliability. In this article, a novel system based on the unmanned aerial vehicle (UAV) is presented to achieve automated aircraft surface inspection efficiently. The hardware is established with a lightweight and low-cost flight platform, on which a sensor containing an inertial measurement unit (IMU) and a camera is equipped for UAV localization. A high-resolution camera is equipped to collect images of fuselage for defect detection. Our inspection framework is mainly composed of two modules: the UAV localization module and the defect detection module. The localization module is designed to estimate the relative pose between the UAV and the aircraft, providing the foundation for image positioning on the aircraft surface. The existing visual–inertial odometry (VIO) approach is adopted to implement the pose estimation. To reduce the large drifts caused by the VIO approach, a novel method is proposed to deploy precalibrated ArUco markers around the aircraft, which serve as external constraints for the VIO objective to realize joint optimization of the camera pose. In addition, an adaptive weighting method is proposed, which takes into consideration the recognition effect of markers to balance the external constraints. The defect detection module aims to detect defects on the fuselage surface from images captured by the high-resolution camera, which is implemented based on deep learning. To address the issue of detection on a few training samples, the transfer learning strategy is exploited to first pretrain the model on a public defect dataset and then fine-tune it on our collected aircraft defect dataset. After detecting the defects, the defective region is reflected on the fuselage surface through the UAV pose on the corresponding frame provided by the localization module, realizing the accurate defect localization. Experiments on both the simulation environment and real data demonstrate the superiority of our proposed external localization module and the effectiveness of the crack detection module.
1 January 2021 in IEEE Robotics and Automation Letters
To enhance the cross-target and cross-scene generalization of target-driven visual navigation based on deep reinforcement learning (RL), we introduce an information-theoretic regularization term into the RL objective. The regularization maximizes the mutual information between navigation actions and visual observation transforms of an agent, thus promoting more informed navigation decisions. This way, the agent models the action-observation dynamics by learning a variational generative model. Based on the model, the agent generates (imagines) the next observation from its current observation and navigation target. This way, the agent learns to understand the causality between navigation actions and the changes in its observations, which allows the agent to predict the next action for navigation by comparing the current and the imagined next observations. Cross-target and cross-scene evaluations on the AI2-THOR framework show that our method attains at least 10% improvement of average success rate over some state-of-the-art models. We further evaluate our model in two real-world settings: navigation in unseen indoor scenes from a discrete Active Vision Dataset (AVD) and continuous real-world environments with a TurtleBot. We demonstrate that our navigation model is able to successfully achieve navigation tasks in these scenarios. 11 [Online]. Available: https://github.com/wqynew/RL-based-navigation.
10 February 2022 in IEEE Transactions on Instrumentation and Measurement
The 6-D pose estimation is a crucial task in vision-based measurement for robotic manipulation. It becomes a challenging task because of the variety of lighting conditions, cluttered background, occlusion, and texture-less objects. The various lighting conditions and texture-less objects lead to dramatic changes for imaging. In this article, we propose an edge-attention 6-D pose estimation network (EANet) for texture-less objects that achieve the autonomous perception of the edge, which is invariant when the lighting conditions change and objects are texture-less. To achieve this, EANet adopts a multitask learning strategy that introduces edge cues into the pixelwise dense fusion framework. We design a shared-weight edge extractor serving for edge reconstruction and pose estimation simultaneously. The purpose of edge reconstruction is to guide the network to pay more attention to edges, so as to enhance the performance of pose estimation implicitly. The edge cues can further improve the performance of pose refinement. Furthermore, we apply a skip-connection scheme to the edge extractor, merging feature maps of the deep and shallow layers. Due to the lightweight design, EANet obtains an inference time of 20 frames per second (FPS) on a desktop PC with an RTX 2070 GPU, which satisfies the requirement of most real-time applications. Experiments show that our network has a significant effect on texture-less objects and approaches the state-of-the-art performance on both the LINEMOD and the YCB-Video datasets. Finally, we present a PARTS dataset to test real-world performance and apply EANet to bin-picking vision measurement, which demonstrates that our method can provide a complete and robust solution for vision-based pose measurement.
25 April 2019 in IEEE Transactions on Multimedia
3D pose estimation for texture-less objects remains a challenging problem. Previous works either focus on a template matching method to find the nearest template as a candidate, or construct a Hough Forest, which utilizes the offset of patches to vote for the object location and pose. By contrast, in this paper, we propose a comprehensive framework to directly regress 3D poses for the candidates, in which a CNN-based triplet network is trained to extract discriminated features from the binary images. To make the features suitable for the regression task, a pose-guided method and a regression constraint are employed to the constructed triplet network. We show that the constraint reaches the goal of creating the correlation between the features and 3D poses. Once the expected features are obtained, the object pose could be efficiently regressed by training a regression network with a simple structure. For symmetric objects, depth images are treated as an additional channel to feed the triplet network. Experiments on the LineMOD and our own datasets demonstrate our method with high regression precision and efficiency.
30 May 2022 in IEEE Robotics and Automation Letters
We present a novel approach for image-goal navigation, where an agent navigates with a goal image rather than accurate target information, which is more challenging. Our goal is to decouple the learning of navigation goal planning, collision avoidance, and navigation ending prediction, which enables more concentrated learning of each part. This is realized by four different modules. The first module maintains an obstacle map during robot navigation. The second predicts a long-term goal on the real-time map periodically, which can thus convert an image-goal navigation task to several point-goal navigation tasks. To achieve these point-goal navigation tasks, the third module plans collision-free command sets for navigating to these long-term goals. The final module stops the robot properly near the goal image. The four modules are designed or maintained separately, which helps cut down the search time during navigation and improve the generalization to previously unseen real scenes. We evaluate the method in both a simulator and in the real world with a mobile robot. The results in real complex environments show that our method attains at least a 17% increase in navigation success rate and a 23% decrease in navigation collision rate over some state-of-the-art models.