As an essential procedure of data fusion, LiDAR-camera calibration is critical for autonomous vehicles and robot navigation. Most calibration methods require laborious manual work, complicated environmental settings, and specific calibration targets. The targetless methods are based on some complex optimization workflow, which is time-consuming and requires prior information. Convolutional neural networks (CNNs) can regress the six degrees of freedom (6-DOF) extrinsic parameters from raw LiDAR and image data. However, these CNN-based methods just learn the representations of the projected LiDAR and image and ignore the correspondences at different locations. The performances of these CNN-based methods are unsatisfactory and worse than those of non-CNN methods. In this paper, we propose a novel CNN-based LiDAR-camera extrinsic calibration algorithm named CFNet. We first decided that a correlation layer should be used to provide matching capabilities explicitly. Then, we innovatively defined calibration flow to illustrate the deviation of the initial projection from the ground truth. Instead of directly predicting the extrinsic parameters, we utilize CFNet to predict the calibration flow. The efficient Perspective-n-Point (EPnP) algorithm within the RANdom SAmple Consensus (RANSAC) scheme is applied to estimate the extrinsic parameters with 2D–3D correspondences constructed by the calibration flow. Due to its consideration of the geometric information, our proposed method performed better than the state-of-the-art CNN-based methods on the KITTI datasets. Furthermore, we also tested the flexibility of our approach on the KITTI360 datasets.

We proposed a multi-sensor fusion-based localization and scene reconstruction method for a complex dynamic scene. The multi-level fusion between multiple sensors was implemented by fusing data collected from different sensors in different system modules. In the front-end of the system, the camera and the LiDAR assisted each other. The LiDAR point clouds provided 3D information for the feature points in the image. The moving objects elimination method based on the image can remove the points on the moving objects in the LiDAR point clouds for localization accuracy improvement and static 3D scene reconstruction. To further improve the localization accuracy, a combination of visual loop closure detection and LiDAR loop closure detection was utilized to ensure the global consistency of scene reconstruction. At the system’s back-end, the observation model of different sensors was integrated to construct a multiple constraint factor graph with nonlinear optimization to obtain the optimal system states. Experimental results demonstrated that the proposed multi-sensor fusion-based localization and scene reconstruction algorithm could operate robustly in multiple complex dynamic scenes.

Detecting loop closures in dynamic environments is a severe challenge for the simultaneous localization and mapping (SLAM) system. Convolutional neural networks (ConvNet) could provide highlevel and abstract representations extracted directly from images as image descriptors. Some novel ConvNet-based methods have been presented. In dynamic environments, they perform better than the stateof-the-art methods which use hand-crafted features. In this paper, (1) We proposed a flexible loop closure detection workflow based on the holistic representations; (2) In this workflow, a post-processing method is applied to the raw holistic ConvNet representations for redundant information compression; (3) In addition, a compression ratio is introduced in (2) to determine how much information will be retained depending on background change and moving objects. We evaluated our workflow in four open datasets. The experimental results demonstrate that the proposed workflow performs better than many state-of-the-art methods and ConvNet-based approaches.

SLAM in dynamic environments is still a severe challenge for most feature-based SLAM systems. Moving objects will lead to terrible errors in the calculation of frame tracking and local mapping. We propose a novel method for keypoints selection to lower the negative effect brought by moving objects during map construction. To address this challenge, we concentrate on the combination of coarse semantic information and a feature-based SLAM system. In this article, a modified CenterNet object detector is proposed as the moving object detection thread for providing coarse semantic information and 2D location. The modified CenterNet can provide a faster and more accurate prediction. For each frame in a sequence, objects in a scenario will be classified into two motion states, non-static and static, according to the category prediction from the moving object detection thread. Then important processing called semantic data association is presented for motion removal in frame tracking thread and local mapping thread. In this way, the keypoints on the non-static objects will not be chosen for calculation. Finally, on this basis, a modified real-time SLAM system is presented. Experimental results in TUM indoor dataset show that the proposed SLAM system outperforms the state-of-the-art ORB-SLAM2 in dynamic environments. Besides that, our method has a better speed-accuracy trade-off than other deep learning-based SLAM systems in dynamic environments.

LiDAR-based single object tracking plays a key role in intelligent vehicles. Current methods typically follow appearance matching or motion-centric frameworks. However, point clouds are usually sparse and incomplete, providing insufficient appearance information for matching. While the motion-centric framework predicts inter-frame motion of targets instead of performing appearance matching for tracking, it neglects contextual information matching of consecutive frames that is conducive to target motion modeling. In this paper, we propose an elegant and effective framework by leveraging Context Matching to guide motion modeling for accurate Tracking (CMTrack). The novel framework possesses two attractive properties: 1) It incorporates a context matching encoder-decoder network to match contextual information of consecutive frames, fully exploring informative cues relevant to target motion. 2) Benefiting from informative motion cues being modeling, CMTrack allows for accurate prediction of inter-frame motion of targets in a one-stage manner. Extensive experiments are conducted on several widely-adopted datasets, i.e. , KITTI, NuScenes and Waymo Open Dataset. Without bells and whistles, our CMTrack demonstrates competitive tracking accuracy ( e.g. , 87.3% and 69.3% precision on KITTI and NuScenes, respectively) compared to state-of-the-art methods, while running at a high speed of 48 Fps on a single Titan Xp GPU.