The prior monitoring of the road surface conditions provides valuable information to vehicle trajectory planning and active control systems. Road surface perception with vision sensors is an emerging technology that has recently gained much attention. However, there are few reports on an exhaustive technical scheme and application practice for driving assistance. This study first creates a large-scale road surface image dataset containing one million samples with detailed road friction level, material, and unevenness level annotations. A convolutional neural network (CNN) classification model constrained by a combined loss function and adapted optimization strategies is trained on the dataset. Then we propose a decision-level fusion method based on the improved Dempster-Shafer evidence theory to enhance the robustness of the classification algorithm. Finally, the developed models are deployed on the embedded hardware platform. The top-1 accuracy for classifying road surface images reaches 92.05% by the CNN model and 97.50% after fusion. Onboard experiments illustrate the advantage and superiority of the developed technical framework, which has excellent potential in real-vehicle applications for driving assistance.

Although the majority of recent autonomous driving systems concentrate on developing perception methods based on ego-vehicle sensors, there is an overlooked alternative approach that involves leveraging intelligent roadside cameras to help extend the ego-vehicle perception ability beyond the visual range. We discover that most existing monocular 3D object detectors rely on the ego-vehicle prior assumption that the optical axis of the camera is parallel to the ground. However, the roadside camera is installed on a pole with a pitched angle, which makes the existing methods not optimal for roadside scenes. In this paper, we introduce a novel framework for Roadside Monocular 3D object detection with ground-aware embeddings, named MonoGAE. Specifically, the ground plane is a stable and strong prior knowledge due to the fixed installation of cameras in roadside scenarios. In order to reduce the domain gap between the ground geometry information and high-dimensional image features, we employ a supervised training paradigm with a ground plane to predict high-dimensional ground-aware embeddings. These embeddings are subsequently integrated with image features through cross-attention mechanisms. Furthermore, to improve the detector’s robustness to the divergences in cameras’ installation poses, we replace the ground plane depth map with a novel pixel-level refined ground plane equation map. Our approach demonstrates a substantial performance advantage over all previous monocular 3D object detectors on widely recognized 3D detection benchmarks for roadside cameras. The code and pre-trained models will be released soon.

Road surface conditions, especially geometry profiles, enormously affect driving performance of autonomous vehicles. Vision-based online road reconstruction promisingly captures road information in advance. Existing solutions like monocular depth estimation and stereo matching suffer from modest performance. The recent technique of Bird’s-Eye-View (BEV) perception provides immense potential to more reliable and accurate reconstruction. This paper uniformly proposes two simple yet effective models for road elevation reconstruction in BEV named RoadBEV-mono and RoadBEV-stereo, which estimate road elevation with monocular and stereo images, respectively. The former directly fits elevation values based on voxel features queried from image view, while the latter efficiently recognizes road elevation patterns based on BEV volume representing correlation between left and right voxel features. Insightful analyses reveal their consistence and difference with the perspective view. Experiments on real-world dataset verify the models’ effectiveness and superiority. Elevation errors of RoadBEV-mono and RoadBEV-stereo achieve 1.83 cm and 0.50 cm, respectively. Our models are promising for practical road preview, providing essential information for promoting safety and comfort of autonomous vehicles. The code is released at https://github.com/ztsrxh/RoadBEV.

Road unevenness is a significant factor affecting the comprehensive driving performance of vehicles. Monitoring road roughness conditions in advance significantly contributes to ride safety and comfort. However, there are hardly comprehensive implementations for comfort promotion purpose based on LiDAR point clouds. This paper proposes a hierarchical scheme that separately describes road regular and irregular unevenness. Firstly, real-vehicle experiments for collecting point clouds of various uneven roads are conducted with the designed hardware platform. A dense point cloud dataset with road bump or pothole annotations is released. Then, an improved PointNet++ model adapted to irregular unevenness recognition is developed. Comparison experiments with the recent segmentation methods verify the superiority of our model in this task. The mIoU metric on the test set reaches 91.6%, outperforming 3.0% than the original model. Road elevation profiles on tire trajectories are finally constructed. The statistical parameters of regular unevenness and geometric features of the irregular sections are extracted, respectively. The proposed perception scheme for road roughness provides detailed and accurate road condition information. It has great potential in practical applications for road unevenness monitoring and vehicle comfort improvement.

The preview of the road surface states is essential for improving the safety and the ride comfort of autonomous vehicles. The created dataset in this data article consists of 370151 road surface images captured under a wide range of road and weather conditions in China. The original pictures are acquired with a vehicle-mounted camera and then the patches containing only the road surface area are cropped. The friction level, material, and unevenness properties of each road image are annotated in detail. This large-scale dataset is useful for developing vision-based road sensing modules to improve the performance of the driving assistance systems. Also, deep-learning experts can regard this dataset as a comparing benchmark for their algorithms. The dataset is available at [1].