A vision-based target-following guider for a mobile robot is presented in this paper. It consists of three parts: the visual tracking part, the target re-detection part, and the visual servo part. In the visual tracking part, the target contour band (TCB) is proposed for irregular sampling, which can improve the performance of the correlation filter-based methods. In the target re-detection part, potential targets are searched by an off-line trained detector. Then an online training module is used to determine the real target and achieve accurate positioning. The interaction matrix of point features is used in the visual servo part for motion control to maintain the relative pose between the target and the robot. The visual tracking part and the target re-detection part are tested respectively on many videos, which are proved to work well. To show the effectiveness of our method, some state of the art methods are also test. The target-following guider is evaluated on mobile robots. In our experiments, the robot can robustly follow the human target over a long distance, which strongly proved the validity of our target-following guider.

Calibration is an important premise for vision-guided robot system, especially for precise industrial operations on curved surfaces. Most of existing methods focus on one or partial union of errors including assembly error of the robot, relative pose error between the vision unit and the robot, or installation error of end effector, which is still challenging in practice. To solve this problem, this article proposes a new calibration method based on error correction matrix by which different manifestations of each error is unified. As the nonlinear equations formed by these matrices cannot be solved analytically, a numerical optimization solution based on Lie algebra is presented. Specifically, the matrices acquisition is formulated as the problem of minimizing the sum of distance deviations between actual and ideal tool center points. This problem is then solved by differentiating these matrices with a form of left multiplicative perturbation. In this way, a high precision joint calibration with multisource errors is achieved. The proposed method is verified by simulations and experiments.

The manipulating robots receive much attention by offering better services, where object grasping is still challenging especially under background interferences. In this article, a novel two-stream grasping convolutional neural network (CNN) with simultaneous detection and segmentation is proposed. The proposed method is cascaded by an improved simultaneous detection and segmentation network BlitzNet and a two-stream grasping CNN TsGNet. The improved BlitzNet introduces the channel-based attention mechanism, and achieves an improvement of detection accuracy and segmentation accuracy with the combination of the learning of multitask loss weightings and background suppression. Based on the obtained bounding box and the segmentation mask of the target object, the target object is separated from the background, and the corresponding depth map and grayscale map are sent to TsGNet. By adopting depthwise separable convolution and designed global deconvolution network, TsGNet achieves the best grasp detection with only a small amount of network parameters. This best grasp in the pixel coordinate system is converted to a desired 6-D pose for the robot, which drives the manipulator to execute grasping. The proposed method combines a grasping CNN with simultaneous detection and segmentation to achieve the best grasp with a good adaptability to background. With the Cornell grasping dataset, the image-wise accuracy and object-wise accuracy of the proposed TsGNet are 93.13% and 92.99%, respectively. The effectiveness of the proposed method is verified by the experiments.

Existing monocular visual simultaneous localization and mapping (SLAM) mainly focuses on point and line features, and the constraints among line features are not fully explored. In this paper, a multi-feature monocular SLAM with ORB points, lines, and junctions of coplanar lines is proposed for indoor environments. To create 3D junctions of coplanar lines, an adaptive coordinate confidence is designed to describe the coordinate stability of 2D junctions on the image plane. Based on the matched 2D junctions between the current keyframe and other two associated keyframes, a multi-view coplanarity verification is performed for 3D junction creation. Moreover, reprojection verification is conducted to update the observation of 3D junction, which is used to dynamically update the confidence of coplanarity for confidence-based local bundle adjustment. The camera pose is optimized through multiple constraints of ORB points, lines, and junctions of the coplanar lines. As a result, the performance of monocular SLAM is improved. The experiment verification shows the effectiveness of the proposed method.

Few-shot object detection (FSOD) aims to detect novel objects with limited annotated examples. Mainstream methods suffer from the data scarcity of novel classes with insufficient intra-class variations, which makes the trained model biased to base classes. Actually, there are massive unlabeled novel instances in the base dataset and their adequate utilization will enhance the discriminability of model to novel classes. This paper proposes a semi-supervised few-shot object detection method, which utilizes a teacher model and a pre-trained few-shot object detector to guide the learning of a student model through adaptive pseudo labeling. In particular, a class-adaptive threshold filtering (CATF) strategy is designed to deal with the class-imbalance problem of pseudo labels. And for each novel class, the threshold to select valuable pseudo labels is determined by quantile statistics of the confidence score distribution of pseudo labels. Furthermore, the pre-trained detector and the teacher model are associated with the preliminary CATF and in-depth CATF, respectively, and then the pseudo labels from the two-stream CATF are fused to provide supervisions. In this way, the knowledge of these two models is exploited, which improves the quality of pseudo labels. Under these supervisions, the student model is trained and the teacher model is correspondingly updated through parameters sharing, thus forming a positive feedback to improve the performance of both models. Besides, an attention module is integrated to the teacher and student models to enhance the feature representation of novel instances. The validations on PASCAL VOC and MS COCO show the effectiveness of the proposed method.

The capability to grasp the target object is significant for manipulating robotic systems to offer better services, and it is still challenging under occlusion. This paper proposes a novel vision-based grasping method with a SSD-based detector, an image inpainting and recognition network (IRNet), and a deep grasping guidance network (DgGNet). Based on the clustering of point cloud, IRNet with the combination of a three-stage image inpainting network and a recognition network MobileNet v2 is introduced to detect the occluded object that cannot be found by the detector. Then, the best grasp for the object to be grasped is obtained by DgGNet, which provides the guidance of the manipulator movement. The image inpainting is firstly introduced into the object detection of manipulating robotic system where the recognition based on inpainting result improves the robustness to occlusion. Experimental results validate the effectiveness of the proposed method.