17 January 2024 in IEEE Transactions on Intelligent Vehicles
Depth completion aims to recover dense depth maps from sparse depth measurements. It is a fundamental challenge in computer vision that is faced in numerous applications, such as robotics, UAVs, and autonomous driving. With the recent advancement in hardware, millimeter-wave (mmWave) radar technology is now commonly employed in high-level perception tasks of autonomous driving. However, compared to LiDAR point clouds, mmWave radar point clouds tend to be sparse and include several ghost targets. To address these challenges, we propose a semantic-guided network architecture to perform depth diffusion by utilizing the consistent relationship between semantics and depth. Our method first utilizes semantic features from images to extend the height measurements, and then transforms the global depth completion task into a series of category-specific depth diffusion tasks to learn the semantic-depth priors and accommodate radar sparsity. In addition, a gradient-guided Smooth-Edge loss is developed to explicitly constrain the semantic regional consistency and border discontinuity. Meanwhile, a 4D dataset that features Camera, 4D Radar, and LiDAR measurement in diverse scenes is collected to conduct experiments. Extensive experiments demonstrate the superior performance of our proposed method over existing monocular and fusion-based approaches.
21 March 2024 in IEEE Transactions on Intelligent Vehicles
Millimeter-wave (mmWave) radar is widely used in autonomous driving thanks to its robustness under harsh weather conditions. However, compared to LiDAR point clouds, mmWave radar point clouds tend to be sparse and include several “ghost” targets. Existing methods attempt to optimize radar signal processing to generate superior radar point cloud. Nonetheless, these methods are primarily designed for simple scenarios. How to obtain superior radar point clouds for complex autonomous driving scenarios is still unexplored. In this article, we propose a novel point cloud generation network that generates superior radar point clouds by utilizing complementary advantage of radar and camera. Technically, our approach comprises three key steps. First, to obtain precise relationships between neighboring pixels, the mmWave range-azimuth map (RAM) is enhanced to extract the radar affinity matrix. Second, to obtain dense image bird's-eye-view (BEV) features, image features are diffused by learning local details from the radar affinity matrix. Third, to obtain superior radar point clouds, radar features and diffused image features are combined in a unified BEV representation, and optimized by K-nearest-neighbor (KNN) graph. To verify the effectiveness of the proposed methods, we collect a real-world dataset under autonomous driving scenarios to conduct experiments. The results demonstrate that our method outperforms exisiting ones in producing high-quality radar point clouds. And as a consequence, the radar point cloud generated using our methods significantly improve the performance of object detection task.
13 June 2025 in IEEE Transactions on Intelligent Vehicles
Recently, pseudo-point-based 3D object detection framework has demonstrated superior performance and attracted significant attention. Unfortunately, current pseudo point generators, the core of this framework, fail to consider accuracy, robustness, and efficiency simultaneously. To meet these challenges, we propose a novel pseudo point generator named VIL-PPGen, which leverages a visible light camera, an infrared camera and a lidar to specifically cater to the requirements of pseudo-point-based detectors. The proposed VIL-PPGen mainly consists of three modules: Dual-spectrum Depth Completion Module (DDCM), Sparse Cost Volume Module (SCVM) and Adaptive Depth Correction Module (ADCM). The DDCM adopts a dual structure for depth completion on different spectrums to maintain accuracy, the SCVM utilizes sparse operations for cost volume computation to improve efficiency, and the ADCM employs adaptive confidence and offset for correction to reinforce accuracy and robustness. Ultimately, we can acquire high-quality pseudo points under all-day lighting conditions, which can directly improve the performance of subsequent detectors. To validate the efficacy of our designs, we construct a dataset from real driving scenarios and conduct extensive experiments. The proposed VIL-PPGen achieves 1.083m MAE (improved by 0.581m) for depth prediction and 53.10% mAP (improved by 3.75%) for multi-class detection in day scenes, while achieving 1.025m MAE (improved by 0.551m) for depth prediction and 55.64% mAP (improved by 3.94%) for multi-class detection in night scenes. It also exhibits excellent real-time performance, achieving a processing speed of 10 FPS. The code is available at https://github.com/Aiuan/VIL-PPGen.
29 April 2026 in IEEE Transactions on Intelligent Vehicles
Recent advances in resolution have positioned 4D radar as a promising sensor for autonomous driving, leading to growing research interest in radar-based perception. Many organizations have released related datasets to explore the potential of radar sensing; however, radar information quality-an important factor affecting perception performance-is often overlooked. This neglect is partly due to a limited understanding of the radar pipeline, particularly among people without a radar signal processing background. Moreover, even when this issue is recognized, closed radar pipelines in existing datasets hinder in-depth investigation of radar information quality. To fill this gap, we propose FlexRadar, a dataset with a fully open radar pipeline for systematic study of radar information quality, along with diverse sensor data to support fusion perception research. In this paper, we provide a comprehensive description and analysis of the dataset. We establish two benchmarks and conduct extensive experiments to demonstrate the effectiveness of the data and the benefits of the open radar pipeline. This work is expected to broaden researchers' perspectives on radar-based perception.
20 April 2026 in IEEE Robotics and Automation Letters
4D millimeter-wave radar plays a critical role in object detection for autonomous driving and robotics under all-weather and all-lighting conditions. Recently, the virtual-point-based approaches have attracted widespread attention due to their ability to address radar data sparsity by complementing the depth of image instance points with the nearest 3D points. However, existing radar-camera fusion methods based on virtual points simply incorporate virtual points into the raw radar points as a form of data augmentation, overlooking the potential of virtual points that have an inherent association with both radar and images. To address these issues, we present a novel radar-camera fusion network, RCVAFusion, for 3D object detection. Specifically, we first design an association branch that employs Object Area Sampling (OAS) and Virtual-Raw Points Depth Lifting (VRPDL). This branch facilitates a deep interaction between radar geometric features and image semantic features through the medium of virtual points to generate an association feature. Then, we introduce the Dual-step Feature Aggregation (DFA) to promote feature fusion from radar, image, and association branches by establishing aggregation priorities based on feature similarity in two steps. Experimental results on the TJ4DRadSet and View-of-Delft (VoD) datasets demonstrate that our method efficiently fuses radar and camera through virtual points and achieves state-of-the-art performance.
26 May 2026 in IEEE Embedded Systems Letters
In mmWave radar embedded systems, conventional constant false alarm rate (CFAR) algorithms struggle to satisfy strict real-time constraints under limited hardware resources while maintaining high detection accuracy, making them difficult to deploy on edge devices. To address this problem, this letter presents a temporal folded reuse CFAR (TFR-CFAR) accelerator using a joint software–hardware co-optimization approach. A lightweight deep neural network (DNN) is adopted and structured for efficient temporal folded reuse. To support embedded implementation, a reusable DNN layer processing unit (RDLPU) and a reconfigurable normalization unit (RNU) are introduced to maximize hardware utilization with minimal arithmetic resources. Implemented on an FPGA, the proposed design achieves a competitive 98.1% detection accuracy while significantly reducing hardware usage (22.3k LUTs, 43 DSPs) and processing time, resulting in the highest accuracy FOM among other methods. The results demonstrate that the proposed architecture achieves an effective balance between detection performance and hardware efficiency for edge embedded systems.