17 January 2024 in IEEE Transactions on Intelligent Vehicles
Depth completion aims to recover dense depth maps from sparse depth measurements. It is a fundamental challenge in computer vision that is faced in numerous applications, such as robotics, UAVs, and autonomous driving. With the recent advancement in hardware, millimeter-wave (mmWave) radar technology is now commonly employed in high-level perception tasks of autonomous driving. However, compared to LiDAR point clouds, mmWave radar point clouds tend to be sparse and include several ghost targets. To address these challenges, we propose a semantic-guided network architecture to perform depth diffusion by utilizing the consistent relationship between semantics and depth. Our method first utilizes semantic features from images to extend the height measurements, and then transforms the global depth completion task into a series of category-specific depth diffusion tasks to learn the semantic-depth priors and accommodate radar sparsity. In addition, a gradient-guided Smooth-Edge loss is developed to explicitly constrain the semantic regional consistency and border discontinuity. Meanwhile, a 4D dataset that features Camera, 4D Radar, and LiDAR measurement in diverse scenes is collected to conduct experiments. Extensive experiments demonstrate the superior performance of our proposed method over existing monocular and fusion-based approaches.
21 March 2024 in IEEE Transactions on Intelligent Vehicles
Millimeter-wave (mmWave) radar is widely used in autonomous driving thanks to its robustness under harsh weather conditions. However, compared to LiDAR point clouds, mmWave radar point clouds tend to be sparse and include several “ghost” targets. Existing methods attempt to optimize radar signal processing to generate superior radar point cloud. Nonetheless, these methods are primarily designed for simple scenarios. How to obtain superior radar point clouds for complex autonomous driving scenarios is still unexplored. In this article, we propose a novel point cloud generation network that generates superior radar point clouds by utilizing complementary advantage of radar and camera. Technically, our approach comprises three key steps. First, to obtain precise relationships between neighboring pixels, the mmWave range-azimuth map (RAM) is enhanced to extract the radar affinity matrix. Second, to obtain dense image bird's-eye-view (BEV) features, image features are diffused by learning local details from the radar affinity matrix. Third, to obtain superior radar point clouds, radar features and diffused image features are combined in a unified BEV representation, and optimized by K-nearest-neighbor (KNN) graph. To verify the effectiveness of the proposed methods, we collect a real-world dataset under autonomous driving scenarios to conduct experiments. The results demonstrate that our method outperforms exisiting ones in producing high-quality radar point clouds. And as a consequence, the radar point cloud generated using our methods significantly improve the performance of object detection task.
13 June 2025 in IEEE Transactions on Intelligent Vehicles
Recently, pseudo-point-based 3D object detection framework has demonstrated superior performance and attracted significant attention. Unfortunately, current pseudo point generators, the core of this framework, fail to consider accuracy, robustness, and efficiency simultaneously. To meet these challenges, we propose a novel pseudo point generator named VIL-PPGen, which leverages a visible light camera, an infrared camera and a lidar to specifically cater to the requirements of pseudo-point-based detectors. The proposed VIL-PPGen mainly consists of three modules: Dual-spectrum Depth Completion Module (DDCM), Sparse Cost Volume Module (SCVM) and Adaptive Depth Correction Module (ADCM). The DDCM adopts a dual structure for depth completion on different spectrums to maintain accuracy, the SCVM utilizes sparse operations for cost volume computation to improve efficiency, and the ADCM employs adaptive confidence and offset for correction to reinforce accuracy and robustness. Ultimately, we can acquire high-quality pseudo points under all-day lighting conditions, which can directly improve the performance of subsequent detectors. To validate the efficacy of our designs, we construct a dataset from real driving scenarios and conduct extensive experiments. The proposed VIL-PPGen achieves 1.083m MAE (improved by 0.581m) for depth prediction and 53.10% mAP (improved by 3.75%) for multi-class detection in day scenes, while achieving 1.025m MAE (improved by 0.551m) for depth prediction and 55.64% mAP (improved by 3.94%) for multi-class detection in night scenes. It also exhibits excellent real-time performance, achieving a processing speed of 10 FPS. The code is available at https://github.com/Aiuan/VIL-PPGen.
29 April 2026 in IEEE Transactions on Intelligent Vehicles
Recent advances in resolution have positioned 4D radar as a promising sensor for autonomous driving, leading to growing research interest in radar-based perception. Many organizations have released related datasets to explore the potential of radar sensing; however, radar information quality-an important factor affecting perception performance-is often overlooked. This neglect is partly due to a limited understanding of the radar pipeline, particularly among people without a radar signal processing background. Moreover, even when this issue is recognized, closed radar pipelines in existing datasets hinder in-depth investigation of radar information quality. To fill this gap, we propose FlexRadar, a dataset with a fully open radar pipeline for systematic study of radar information quality, along with diverse sensor data to support fusion perception research. In this paper, we provide a comprehensive description and analysis of the dataset. We establish two benchmarks and conduct extensive experiments to demonstrate the effectiveness of the data and the benefits of the open radar pipeline. This work is expected to broaden researchers' perspectives on radar-based perception.