29 May 2023 in IEEE Transactions on Instrumentation and Measurement
Camera and millimeter-wave (MMW) radar fusion is essential for accurate and robust autonomous driving systems. With the advancement of radar technology, next-generation high-resolution automotive radar, i.e., 4D radar, has emerged. In addition to the target range, azimuth, and Doppler velocity measurements of traditional radar, 4D radar provides elevation measurement to create a denser “point cloud.” In this study, we propose a camera and 4D radar fusion network called RCFusion, which achieves multimodal feature fusion under a unified bird’s-eye view (BEV) space to accomplish 3D object detection tasks. In the camera stream, multi-scale feature maps are obtained by the image backbone and feature pyramid network; they are then converted into orthographic feature maps by an orthographic feature transform. Next, enhanced and fine-grained image BEV features are obtained via a designed shared attention encoder. Meanwhile, in the 4D radar stream, a newly designed component named Radar PillarNet efficiently encodes the radar features to generate radar pseudo-images, which are fed into the point cloud backbone to create radar BEV features. An interactive attention module is proposed for the fusion stage, which outputs a valid fusion of the two-modal BEV features. Finally, a generic detection head predicts the object classes and locations. The proposed RCFusion is validated on the TJ4DRadSet and View-of-Delft datasets. The experimental results and analysis show that the proposed method can effectively fuse camera and 4D radar features to achieve robust detection performance.
26 September 2023 in IEEE Transactions on Intelligent Vehicles
This article introduces a versatile multi-task learning framework (UMT-Net) and an Adaptive Task Weighting (ATW) training method, specifically designed for resource-constrained scenarios that demand parameter-efficient networks. The adaptable UMT-Net architecture includes a global-shared backbone based on an encoder, task-specific self-attention modules, inter-task joint-attention fusion modules, and feature-aggregating decoders. The ATW technique accounts for both short-term variation and long-term statistics of task losses, leading to a more stable training process. Extensive experiments on CityScapes and NYUv2 datasets reveal that UMT-Net outperforms baseline methods while requiring fewer computations, model parameters, and inference latency. In addition, we also conducted experiments on the autonomous driving dataset BDD100K and achieved state-of-the-art performance. Furthermore, we deployed the model and carried out tests for generalization in real-world scenarios. Finally, our network architecture possesses the capability to be designed as a compact model with much fewer parameters, computational requirements, and inference time, while maintaining competitive performance, making it suitable for deployment on mobile devices.