This letter presents a real-time monocular visual-inertial simultaneous localization and mapping with point-line fusion and parallel-line fusion. The corner and line features provide plenty of information about object structures. In the 2D image plane, such corner and line features have a positional similarity. The corner feature represents an endpoint of an object's edge, and the line feature represents a straight edge. We propose two novel methods for fusing corner and line features to improve localization accuracy. The first method is for fusing corner and line features. Using the positional similarity of corner and line features, we search the relationship between two features by utilizing the proposed point-line coupled residual. The second method is for fusing parallel 3D lines. First, we search for line features clustered based on a vanishing point. Then, the outliers in the parallel 3D lines are removed using the proposed consistency check during the multi-view line clustering. Finally, the parallel 3D lines are used in the proposed parallel 3D line residual. Experimental results show that real-time localization accuracy is improved when two proposed residuals are integrated with the sliding-window optimization. The proposed PLF-VINS is compared with other state-of-the-art algorithms using the public EuRoC dataset.

The purpose of an excavator is to dig up materials and load them onto heavy-duty dump trucks. Typically, an excavator is positioned at the rear of the dump truck when loading. In order to automate this process, this paper proposes a system that employs a combined stereo camera and two LiDAR sensors to determine the three-dimensional (3D) position of the truck’s cargo box and to analyze its loading space. Sparse depth information acquired from the two LiDAR sensors is used to detect points on the door of the cargo box and establish the plane on its rear side. Dense depth information of the cargo box acquired from the stereo camera sensor is projected onto the plane of the box’s rear to estimate its initial 3D position. In the next step, the point cloud sampled along the long shaft of the edge of the cargo box is used as the input of the Iterative Closest Point algorithm to calculate a more accurate cargo box position. The data collected from the stereo camera are then used to determine the 3D position of the cargo box and provide an estimate of the volume of the load along with the 3D position of the loading space to the excavator. In order to demonstrate the efficiency of the proposed method, a mock-up of a heavy-duty truck cargo box was created, and the volume of the load in the cargo box was analyzed.

This paper proposes an automatic slag removal path estimation in a ladle image using a deep neural network. Slag removal is a hazardous task in steel production, and it is conducted manually by experienced human operators. To automate the slag removal task using a robotic machine, a slag removal path in a ladle image must be determined to control the motion of the machine. We propose a novel image structure called Slag Distribution Image(SDI) and a Slag Removal Path Estimation Network(SRPENet) to estimate the slag removal path. SDI contains the slag distribution and the ladle boundary information. SRPENet is designed as a multi-task learning architecture to perform two tasks: the first task is to estimate the control point of the slag removal path and the second task is to estimate the goodness-score of the control points. The structure of SDI is designed to yield efficient performance in SRPENet. The ground truth and learning data of slag removal path are collected along the working trajectories of human experts. The final removal path is determined by post-processing of the outputs of SRPENet. In experiments using real ladle images, the amount of removed slag and the mechanical energy efficiency are evaluated and compared with the ground truth to present qualitative and quantitative analysis. In addition, two ablation studies are presented. The first study is using SDI or RGB image as the input of SRPENet. The second study is the presence or absence of the smooth term in the SRPENet’s loss function. The proposed method shows 119.83% and 96.2% performance compared to ground truth in the amount of removed slag and mechanical energy efficiency, respectively. Our method runs in real-time with about 53.168 msec/frame.