Due to the increase of signal bandwidth, automotive radar can receive multiple scatting centers from the target of interest. This is typically described by an extended target model. However, existing methods for extended target detection and tracking often exhibit subpar performance because they struggle to adapt to the hybrid-scale and time-varying texture feature exhibited by moving traffic participants. Therefore, this work presents a unified framework hybrid-scale spatio-temporal transformer for joint detection and tracking of hybrid-scale extended targets. It consists of three progressive stages: 1) hybrid-scale embedding network (HEN); 2) spatio-temporal transformer; and 3) Wasserstein loss. The HEN extracts the features of hybrid-scale targets by parallel multiple convolution kernels of different scales. Then, the spatio-temporal transformer locates targets by integrating these spatial-temporal features across the time. It can allow the target’s hybrid-scale and time-varying texture feature changes incurred through the movement, while eliminating the need for cross frame target association, thus achieving robust detection and tracking. The Wasserstein loss metric is designed to assist the model in optimizing hybrid-scale target location proficiency, especially for small targets. With intensive experiments on simulated datasets and public real datasets, such as RAMP-convolutional neural network and RADDet, the proposed model achieves better detection and tracking performance than existing methods with different noise added, and shows strong robustness in various traffic scenarios.
Association is crucial but easy to be neglected in radar and vision object-level fusion. Measurement uncertainty, consisting of complementary random errors of heterogeneous sensors and unpredictable deviation error due to monocular vision ranging, impedes a powerful radar and vision association. This paper proposes a robust radar and vision association method for measurement uncertainty and loss. First, radar and vision measurements are obtained by data preprocessing. Then, the cost matrix is initialized using the weighted Euclidean distance based on measurement random errors and road lane constraints. Next, the initial cost matrix is adaptively modified based on vision measurement calibration model to respond to vision deviation error. Finally, Hungarian algorithm is enhanced to tackled the unbalanced assignment problem caused by measurement loss. Experiments under different environmental conditions (clear weather, rain, and nighttime) show the average total correct association rate of the proposed method is at least 5.24% higher than comparative methods, indicating a more reliable association performance, especially in the distant region.
15 October 2025 in IEEE Transactions on Intelligent Transportation Systems
Radar and vision fusion in autonomous driving has gained increasing attention, however, most existing feature-level fusion methods (named as feature map-based fusion) typically map sparse point clouds into dense feature maps, fuse them with visual features, and then apply the combined features to detect sparse objects. These methods face challenges due to the structural disparity between sparse point clouds and dense feature maps, and memory-redundant mapping of point cloud processing. This paper proposes a proposal-guided radar and vision fusion framework for 3D object detection in autonomous driving, where proposals are referred to as potential objects with abundant high dimensional features. Specifically, two frame radar point clouds are first accumulated by deep learning-based motion compensation with supervision. Then, proposals are generated from radar point clouds and vision images. Subsequently, radar and vision fusion is performed on the proposal level using attention mechanism. Finally, objects are detected based on the fused proposal feature. The proposed framework introduces an object-centric feature fusion methodology that skips the common process of point-to-feature map projection, thereby circumventing the inherent structural disparity between sparse point clouds and dense feature maps. Experiments on two different autonomous driving datasets (nuScenes and VoD) demonstrate that our fusion method can outperform most published radar and vision fusion networks. The source code will be released at https://github.com/leehungu/prv
Multi-radar fusion is fundamental for robust, all-weather perception for diverse applications. However, current fusion paradigms face structural and computational bottlenecks. Traditional statistical frameworks suffer from an explosion of dimensional calculation, where computational complexity scales with the number of active sensor nodes. Concurrently, existing statistical and deep learning fusion models exhibit systemic brittleness; their rigid topological binding to predefined sensor counts leads to a drop in performance during sensor dropouts. Furthermore, generic attention mechanisms suffer a phenomenological mismatch with radar signals, neglecting the spatial features of radar targets and leading to false alarms. To overcome these limitations, we propose RadarsBEV, a scalable end-to-end multi-radar detection framework. By decoupling per-sensor feature extraction from the central spatial fusion process, RadarsBEV achieves permutation invariance. This design breaks the scalability limit and enables graceful degradation utilizing residual nodes without system downtime. Crucially, we introduce a physics-aware Gaussian cross-attention mechanism. By guiding sparse feature sampling through predicted two-dimensional Gaussian target geometry, this mechanism decouples attention weights from clutter signal. Extensive experiments on high-fidelity simulations and real-world datasets demonstrate that RadarsBEV achieves better detection performance. Notably, the framework exhibits robust configuration zero-shot generalization, adapting to entirely unseen spatial layouts and degraded operational environments without fine-tuning.