Automation of the bin picking task with robots entails the key step of pose estimation, which identifies and locates objects so that the robot can pick and manipulate the object in an accurate and reliable way. This paper proposes a novel point pair feature-based descriptor named Boundary-to-Boundary-using-Tangent-Line (B2B-TL) to estimate the pose of industrial parts including some parts whose point clouds lack key details, for example, the point cloud of the ridges of a part. The proposed descriptor utilizes the 3D point cloud data and 2D image data of the scene simultaneously, and the 2D image data could compensate the missing key details of the point cloud. Based on the descriptor B2B-TL, Multiple Edge Appearance Models (MEAM), a method using multiple models to describe the target object, is proposed to increase the recognition rate and reduce the computation time. A novel pipeline of an online computation process is presented to take advantage of B2B-TL and MEAM. Our algorithm is evaluated against synthetic and real scenes and implemented in a bin picking system. The experimental results show that our method is sufficiently accurate for a robot to grasp industrial parts and is fast enough to be used in a real factory environment.

We propose a CNN based visual servoing scheme for precise positioning of an eye-to-hand manipulator in which the control input of a robot is calculated directly from images by a neural network. In this paper, we propose Difference of Encoded Features driven Interaction matrix Network (DEFINet), a new convolutional neural network (CNN), for eye-to-hand visual servoing. DEFINet estimates a relative pose between desired and current end-effector from desired and current images captured by an eye-to-hand camera. DEFINet includes two branches of the same CNN that share weights and encode target and current images, which is inspired by the architecture of Siamese network. Regression of the relative pose from the difference of the encoded target and current image features leads to a high positioning accuracy of visual servoing using DEFINet. The training dataset is generated from sample data collected by operating a manipulator randomly in task space. The performance of the proposed visual servoing is evaluated through numerical simulation and experiments using a six-DOF industrial manipulator in a real environment. Both simulation and experimental results show the effectiveness of the proposed method.

3D Instance segmentation is a fundamental task in computer vision. Effective segmentation plays an important role in robotic tasks, augmented reality, autonomous driving, etc. With the ascendancy of convolutional neural networks in 2D image processing, the use of deep learning methods to segment 3D point clouds receives much attention. A great convergence of training loss often requires a large amount of human-annotated data, while making such a 3D dataset is time-consuming. This paper proposes a method for training convolutional neural networks to predict instance segmentation results using synthetic data. The proposed method is based on the SGPN framework. We replaced the original feature extractor with “dynamic graph convolutional neural networks” that learned how to extract local geometric features and proposed a simple and effective loss function, making the network more focused on hard examples. We experimentally proved that the proposed method significantly outperforms the state-of-the-art method in both Stanford 3D Indoor Semantics Dataset and our datasets.

Pose estimation that locates objects in a bin is necessary for a robotic bin picking system. Although many algorithms have shown high performance in pose estimation, most algorithms estimate the poses of objects regardless of their occlusion. This can reduce the success rate in picking up the object. To resolve this issue, we propose a novel pipeline that estimates a pose only for occlusion-free objects based on point pair feature-based pose estimation with multiple edge appearance model (PPF-MEAM). The proposed method detects occlusion-free objects in the 2D image captured by a camera with a convolutional neural network framework. Next, corresponding point clouds of occlusion-free objects need to be extracted by using their locations in the 2D image. we propose a robust extraction method that finds the 3D points corresponding to image pixels in the 2D image to reduce the effect of the calibration errors between the camera and 3D sensor. The point cloud of the occlusion-free objects is finally input into a pipeline of PPF-MEAM to estimate the pose of the object. The experiment results prove that the proposed method is about 50% faster 30% higher in terms of pose estimation success rate compared with the original PPF. Moreover, it increases the success rate of picking tasks compared with the original PPF-MEAM.

This paper proposes a fast and highly accurate prediction method for the reaching motions performed by a worker in cooperative work with a robot, called fast prediction of reaching motion (FPREMO). Cooperative work is permitted by a 2011 ISO revision under the condition that risk assessment has been performed. To increase production effectivity, it is essential to predict the worker’s movement and control the robot based on the prediction. This paper focuses on the movement of a worker’s hand, called the reaching motion, which is relevant to many types of assembling tasks. The most common existing methods require attaching markers to the workers or installing three-dimensional (3D) sensors in front of the workers since these methods require the real-time estimation of skeleton models of the workers. These requirements lead to difficulties in introducing prediction methods at production sites. To solve this problem, we propose a prediction method for the reaching motion that neither requires the attachment of markers nor limits the placement of the 3D sensor. The proposed method first computes a feature vector of the reaching motion and then performs the prediction using random forest with the computed feature vector as input. Experimental results show that the proposed method can predict the reaching motion from the initial 50% of the movement of the worker with more than 80% accuracy in less than 5.2 ms.

3D keypoint is widely used for object recognition and pose estimation with a 3D point cloud since it has robustness against measurement noise and occlusion. Many sets of 3D keypoints’ definitions and the corresponding detection methods of 3D keypoints have been proposed. It is essentially important to select suitable 3D keypoint depending on a target object, scene situation, and 3D measurement system. While there have been a lot of useful methods for detecting 3D keypoints, detecting time is an issue, especially for real-time applications, such as robot vision. The detecting of 3D keypoints tends to entail a trade-off between computation time and the robustness of the 3D keypoint. To solve these problems, we propose a FAst Detection Algorithm for 3D keypoints named FADA-3K. A user can select one of the suitable 3D keypoints and the corresponding detection method which has been already proposed since FADA-3K can be implemented as an add-on of the existing detection methods. Numerical experiments show that FADA-3K can achieve about nine times faster detection than conventional approaches to detecting 3D keypoints.