Road detection is an important task in autonomous navigation systems. In this paper, we propose a road detection method via a LiDAR-camera fusion strategy to exploit both the range and color information. The whole system consists of three parts. In the LiDAR based part, we transform the discrete 3D LiDAR point clouds to continuous 2D LiDAR range images and propose a distance-aware height-difference based scanning approach to get the road estimations quickly. In the camera based part, we apply a light-weight transfer learning based road segmentation network. In the LiDAR-camera fusion part, we transform the detection results from LiDAR and camera to dense and binary ones to solve the data imbalance problem and fuse them in a multi-modal conditional random field (MM-CRF) framework. Experiments show that the proposed MM-CRF fusion method can operate in real-time and achieve competitive performance compared with the state-of-the-art road detection algorithms on the KITTI-Road benchmark.

In this paper, we propose to fuse the geometric information of a 3-D LiDAR and a monocular camera to detect the urban road region ahead of an autonomous vehicle. Our method takes advantage of both the high definition of 3-D LiDAR data and the continuity of road in image representation. First, we obtain an efficient representation of LiDAR data and an organized 2-D inverse depth map, by projecting the 3-D LiDAR points onto the camera's image plane. Through the new representation, we can acquire the intermediate representations of road scenes by extracting the vertical and horizontal histograms of the normalized inverse depth. The approximate road regions can be quickly estimated with both histogram-based schemes. To accurately find the road area, we propose a row and column scanning strategy in the approximate road region to refine the detected road area. We have carried out experiments on the public KITTI-Road benchmark, and have achieved one of the best performances among the LiDAR-based road detection methods without learning procedure.

The visual-inertial SLAM, which integrates data from monocular or stereo cameras, has garnered significant attention and development. The RGB-D camera, capable of capturing both color and depth images simultaneously, can perceive a comprehensive view of the surroundings. To fully leverage two types of measurement information from the RGB-D camera and IMU sensor for accurate pose estimation, we propose a new visual-inertial SLAM algorithm, VID-SLAM, that effectively couples the RGB-D camera with the IMU. In our proposal, we first develop an adaptive point feature detection approach that rapidly detects and tracks sufficient point features. This approach utilizes adaptive non-maximum suppression and the Kd-Tree algorithm to ensure a homogeneous distribution of point features. Secondly, we incorporate line features into the pose estimation module of the SLAM algorithm. By screening line features based on the geometric properties of vanishing points, we ensure that the detected lines align with the edges of scene objects as early as possible. Beyond the 2D re-projection error of line features, we introduce a new error term that leverages the geometric constraints of plane normal vectors formed by matched line features and the optical center of the RGB-D camera. Furthermore, we estimate the pose of the RGB-D camera by loosely coupling point-line visual features with IMU pre-integration measurements. In the back end of VID-SLAM, we tightly couple the point-line feature error model with the IMU pre-integration to jointly optimize the camera pose. Extensive qualitative and quantitative comparisons demonstrate that our VID-SLAM algorithm achieves robust performance and comparable accuracy.

Over recent years, there has been an increase in research interest regarding depth estimation using multiple-spectrum images from Visible-Light (VIS) and Thermal-Infrared (TIR) cameras. Depth estimation with multi-spectrum images has the potential for more reliable depth estimation under varying illumination conditions. However, most of the existing multi-spectrum depth estimation methods rely on the supervision of ground-truth depth information or other additional complex labels or equipment, resulting in limited convenience for practical application. To address these challenges, we propose a new unsupervised all-day depth estimation framework with multi-spectrum stereo images, specifically, a thermal image (acting as the right view) and a visible-light image (acting as the left view), and they are denoted as a single frame of multi-spectrum stereo sequences. The network uses multiple frames of multi-spectrum stereo images as input during training and a single frame of multi-spectrum stereo images as input for testing. We improve the depth estimation accuracy by combining the intra-spectrum temporal consistency and the cross-spectrum spatial consistency. A unique self-teacher structure is integrated to augment the quality of nighttime depth estimation using daytime image data. Simultaneously, we release a large multi-spectrum stereo dataset, which includes TIR-VIS stereo images collected during the day and night. Experimental results reveal that our method achieves a significant improvement in all-day depth estimation by effectively utilizing multi-spectrum stereo images.

Dense semantic scene understanding of the surrounding environment in top-view is a crucial task for autonomous vehicles. Recent LiDAR-based semantic perception works mainly focus on point-wise predictions of the LiDAR points instead of dense predictions of the environment, making them not appropriate for path-planning tasks. Pillar and voxel representations can achieve dense predictions, but the generation of data representation and data processing are usually time-consuming. In this paper, we propose a top-view semantic completion network to produce accurate dense grid-wise predictions with real-time performance. Specifically, we propose an online distillation strategy, consisting of two parts: a student model using 2D range-view and top-view representations, and a teacher model using range-view, top-view, and voxel representations. To realize information transfer between different representations, we propose a cross-view association (CVA) module, by which the range-view features and 3D voxel features are converted into the ones in the top-view. The proposed method can avoid the difficulty of direct dense semantic segmentation in the top-view, with the point-wise sparse semantic segmentation module acting as a guide for the dense grid-wise semantic completion in a semantic-completion way. It can also alleviate the computational complexity by using only the voxel representation and 3D convolution in the teacher model. The experimental results on the SemanticKITTI dataset (46.4% mIoU) and nuScenes-LidarSeg dataset (47.3% mIoU) demonstrate the effectiveness of the proposed sparse guidance and online distillation strategies.