Driven by deep learning techniques, perception technology in autonomous driving has developed rapidly in recent years, enabling vehicles to accurately detect and interpret surrounding environment for safe and efficient navigation. To achieve accurate and robust perception capabilities, autonomous vehicles are often equipped with multiple sensors, making sensor fusion a crucial part of the perception system. Among these fused sensors, radars and cameras enable a complementary and cost-effective perception of the surrounding environment regardless of lighting and weather conditions. This review aims to provide a comprehensive guideline for radar-camera fusion, particularly concentrating on perception tasks related to object detection and semantic segmentation. Based on the principles of the radar and camera sensors, we delve into the data processing process and representations, followed by an in-depth analysis and summary of radar-camera fusion datasets. In the review of methodologies in radar-camera fusion, we address interrogative questions, including “why to fuse”, “what to fuse”, “where to fuse”, “when to fuse”, and “how to fuse”, subsequently discussing various challenges and potential research directions within this domain. To ease the retrieval and comparison of datasets and fusion methods, we also provide an interactive website: https://radar-camera-fusion.github.io .
26 June 2024 in IEEE Transactions on Intelligent Transportation Systems
Autonomous driving on water surfaces plays an essential role in executing hazardous and time-consuming missions, such as maritime surveillance, survivor rescue, environmental monitoring, hydrography mapping and waste cleaning. This work presents WaterScenes, the first multi-task 4D radar-camera fusion dataset for autonomous driving on water surfaces. Equipped with a 4D radar and a monocular camera, our Unmanned Surface Vehicle (USV) proffers all-weather solutions for discerning object-related information, including color, shape, texture, range, velocity, azimuth, and elevation. Focusing on typical static and dynamic objects on water surfaces, we label the camera images and radar point clouds at pixel-level and point-level, respectively. In addition to basic perception tasks, such as object detection, instance segmentation and semantic segmentation, we also provide annotations for free-space segmentation and waterline segmentation. Leveraging the multi-task and multi-modal data, we conduct benchmark experiments on the uni-modality of radar and camera, as well as the fused modalities. Experimental results demonstrate that 4D radar-camera fusion can considerably improve the accuracy and robustness of perception on water surfaces, especially in adverse lighting and weather conditions. WaterScenes dataset is public on https://waterscenes.github.io.
23 March 2021 in International Journal of Pavement Engineering
A new crack detection approach based on local binary patterns (LBP) with support vector machine (SVM) was proposed in this paper. The propsed algorithm can extract the LBP feature from each frame of the video taken from the road. Then, the dimension of the LBP feature spaces can be reduced by Principal Component Analysis(PCA). The simplified samples are trained to be decided the type of crack using Support Vector Machine(SVM). In order to reflect the directional imformation in detail, the LBP processed image is devided into nine sub-blocks. In this paper, driving tests were performed 10 times and 12,000 image data were applied to the proposed algorithm. The average accuracy of the proposed algorithm with sub-blocks is 91.91%, which is about 6.6% higher than the algorithm without sub-blocks. The LBP-PCA with SVM applying sub-blocks reflects the directional information of the crack so that it has high accuracy of 89.41% and 88.24%, especially in transverse and longitudinal cracks. In the performance analysis of different crack classifiers, the F-Measure, which considered balance between the precision and the recall, of alligator cracks classifier was the highest at 0.7601 and hence crack detection performance is higher than others.
7 April 2025 in IEEE Transactions on Intelligent Transportation Systems
With the rapid advancements of sensor technology and deep learning, autonomous driving systems are providing safe and efficient access to intelligent vehicles as well as intelligent transportation. Among these equipped sensors, the radar sensor plays a crucial role in providing robust perception information in diverse environmental conditions. This review focuses on exploring different radar data representations utilized in autonomous driving systems. Firstly, we introduce the capabilities and limitations of the radar sensor by examining the working principles of radar perception and signal processing of radar measurements. Then, we delve into the generation process of five radar representations, including the ADC signal, radar tensor, point cloud, grid map, and micro-Doppler signature. For each radar representation, we examine the related datasets, methods, advantages and limitations. Furthermore, we discuss the challenges faced in these data representations and propose potential research directions. Above all, this comprehensive review offers an in-depth insight into how these representations enhance autonomous system capabilities, providing guidance for radar perception researchers. To facilitate retrieval and comparison of different data representations, datasets and methods, we provide an interactive website at https://radar-camera-fusion.github.io/radar.
Automatic damage detection using deep learning warrants an extensive data source that captures complex pavement conditions. This paper proposes a thermal-RGB fusion image-based pavement damage detection model, wherein the fused RGB-thermal image is formed through multi-source sensor information to achieve fast and accurate defect detection including complex pavement conditions. The proposed method uses pre-trained EfficientNet B4 as the backbone architecture and generates an argument dataset (containing non-uniform illumination, camera noise, and scales of thermal images too) to achieve high pavement damage detection accuracy. This paper tests separately the performance of different input data (RGB, thermal, MSX, and fused image) to test the influence of input data and network on the detection results. The results proved that the fused image’s damage detection accuracy can be as high as 98.34% and by using the dataset after augmentation, the detection model deems to be more stable to achieve 98.35% precision, 98.34% recall, and 98.34% F1-score.
In this paper, field construction data from the Singapore Metro Line project were used to study the mapping relationship and establish the prediction model between TBM operation data and the ground condition ahead of the excavation face. The study presents a multi-classifier competition mechanism to construct ten separate classifiers, including logistic regression, support vector machine, random forest, extremely randomized trees, adaptive boosting machine, extreme gradient boosting (Xgboost), light gradient boosting (LightGBM), categorical boosting, long short-term memory and convolutional neural network. The acquired data were used to select 28 key TBM operating parameters by a correlation-based feature selection method, and the selected parameters in the stabilization phase after removing the outliers were calculated as the input to the classifier, and a relatively balanced training set was obtained by the synthetic minority oversampling technique. The hyperparameters of each classifier were optimized using tree Parzen estimator Bayesian optimization. The prediction results show that LightGBM presents the best results among ten different machine and deep learning algorithms with an accuracy of 96.22%, precision of 96.94%, recall of 97.33% and F1-score of 97.33%. In addition, the effect of the input parameters of the LightGBM model on the prediction accuracy of the model was analyzed using Shapley additive explanations, and the effect of sample imbalance on the prediction performance was discussed.