Environmental perception with the multi-modal fusion is crucial in autonomous driving to increase accuracy, completeness, and robustness. This paper focuses on utilizing millimeter-wave (MMW) radar and camera sensor fusion for 3D object detection. A novel method that realizes the feature-level fusion under the bird's-eye view (BEV) for a better feature representation is proposed. Firstly, radar points are augmented with temporal accumulation and sent to a spatial-temporal encoder for radar feature extraction. Meanwhile, multi-scale image 2D features which adapt to various spatial scales are obtained by image backbone and neck model. Then, image features are transformed to BEV with the designed view transformer. In addition, this work fuses the multi-modal features with a two-stage fusion model called point-fusion and ROI-fusion, respectively. Finally, a detection head regresses objects category and 3D locations. Experimental results demonstrate that the proposed method realizes the state-of-the-art (SOTA) performance under the most crucial detection metrics–mean average precision (mAP) and nuScenes detection score (NDS) on the challenging nuScenes dataset.

With the rapid development of automated vehicles (AVs), more and more demands are proposed towards environmental perception. Among the commonly used sensors, MMW radar plays an important role due to its low cost, adaptability In different weather, and motion detection capability. Radar can provide different data types to satisfy requirements for various levels of autonomous driving. The objective of this study is to present an overview of the state-of-the-art radar-based technologies applied In AVs. Although several published research papers focus on MMW Radars for intelligent vehicles, no general survey on deep learning applied In radar data for autonomous vehicles exists. Therefore, we try to provide related survey In this paper. First, we introduce models and representations from millimeter-wave (MMW) radar data. Secondly, we present radar-based applications used on AVs. For low-level automated driving, radar data have been widely used In advanced driving-assistance systems (ADAS). For high-level automated driving, radar data is used In object detection, object tracking, motion prediction, and self-localization. Finally, we discuss the remaining challenges and future development direction of related studies.

Accurate trajectory prediction of surrounding vehicles is vital for automated vehicles to achieve high-level driving safety in complex situations. However, most state-of-the-art approaches for multi-vehicle trajectory prediction ignore vehicle motion uncertainty caused by different driving styles. Moreover, the interrelationship between the vehicle and the environment is seldom considered. To address the above problems, this paper proposes a driving risk map-integrated deep learning (DRM-DL) method for interactive trajectory prediction of surrounding vehicles, which comprehensively considers the motion uncertainty, trajectory intention uncertainty and interactions among vehicles, lane lines and road boundaries. Specifically, we adopt a conditional variational autoencoder (CVAE) to generate the candidate trajectories, in which the motion uncertainty is considered using a conditional Gaussian distribution. Furthermore, a driving risk map is constructed to realize a unified and interpretable representation of vehicle-vehicle and vehicle-environment interactions. The probability of each candidate trajectory is assigned using a trajectory probability model and a random selection is adopted to select a guided trajectory, which simulates the driver’s trajectory intention uncertainty. Finally, a relearning module is designed to obtain the precise trajectory prediction for surrounding vehicles. The proposed method is evaluated on the HighD dataset, and the results demonstrate a more accurate and reliable trajectory prediction for surrounding vehicles compared with state-of-the-art methods.

The fusion of light detection and ranging (LiDAR) and camera data is a promising approach to improve the environmental perception and recognition for intelligent vehicles because of the combination of depth and color information. One of the difficulties in achieving the fusion is the accurate alignment of the 3-D points with the image pixels. Current methods of data alignment involve the steps of estimating the camera intrinsic parameters and developing a transformation matrix between the camera and LiDAR frame. The drawback of these methods is the accumulation of errors during the calculation of the camera intrinsic parameters and the transformation matrix. In order to improve the data alignment accuracy, we propose a novel algorithm that directly calculates the alignment between the 3-D points and the pixels without the need for camera parameters and calibration of the coordinate transformation matrix. We call the proposed method the pixel and 3-D point alignment (PPA) method. The alignment procedure is achieved by using the extracted corresponding points. First, we calculate a linear alignment matrix without considering the image distortion; and second, we optimize the parameters using the maximum likelihood estimation to consider the camera distortion. Simulation and experimental results indicate that the PPA method is able to align the 3-D points in LiDAR frame with the pixels in image frame with higher accuracy and increased robustness against noise in calibration process than comparable state-of-the-art methods.

The grid-centric perception is a crucial field for mobile robot perception and navigation. Nonetheless, the grid-centric perception is less prevalent than object-centric perception as autonomous vehicles need to accurately perceive highly dynamic, large-scale traffic scenarios, and the complexity and computational costs of grid-centric perception are high. In recent years, the rapid development of deep learning techniques and hardware provides fresh insights into the evolution of grid-centric perception. The fundamental difference between grid-centric and object-centric pipeline lies in that grid-centric perception follows a geometry-first paradigm which is more robust to the open-world driving scenarios with endless long-tailed semantically unknown obstacles. Recent research demonstrates the great advantages of grid-centric perception, such as comprehensive fine-grained environmental representation, greater robustness to occlusion and irregular-shaped objects, better ground estimation, and safer planning policies. There is also a growing trend that the capacity of occupancy networks is greatly expanded to 4-D scene perception and prediction, and the latest techniques are highly related to new research topics, such as 4-D occupancy forecasting, generative artificial intelligence (GenAI), and world models in the field of autonomous driving. Given the lack of current surveys for this rapidly expanding field, we present a hierarchically structured review of grid-centric perception for autonomous vehicles. We organize previous and current knowledge of occupancy grid techniques along the main vein from 2-D bird-eye view (BEV) grids to 3-D occupancy to 4-D occupancy forecasting. We additionally summarize label-efficient occupancy learning and the role of grid-centric perception in driving systems. Finally, we present a summary of the current research trend and provide future outlooks.