Radar and camera information fusion sensing methods are used to solve the inherent shortcomings of the single sensor in severe weather. Our fusion scheme uses radar as the main hardware and camera as the auxiliary hardware framework. At the same time, the Mahalanobis distance is used to match the observed values of the target sequence. Data fusion based on the joint probability function method. Moreover, the algorithm was tested using actual sensor data collected from a vehicle, performing real-time environment perception. The test results show that radar and camera fusion algorithms perform better than single sensor environmental perception in severe weather, which can effectively reduce the missed detection rate of autonomous vehicle environment perception in severe weather. The fusion algorithm improves the robustness of the environment perception system and provides accurate environment perception information for the decision-making system and control system of autonomous vehicles.

The use of object detection algorithms has become extremely important in autonomous vehicles. Object detection at high accuracy and a fast inference speed is essential for safe autonomous driving. Therefore, the balance between the effectiveness and efficiency of the object detector must be considered. This article proposes a one-stage object detection framework for improving the detection accuracy while supporting a true real-time operation based on the YOLOv4. The backbone network in the proposed framework is the CSPDarknet53_dcn(P). The last output layer in the CSPDarknet53 is replaced with deformable convolution to improve the detection accuracy. In order to perform feature fusion, a new feature fusion module PAN++ is designed and five scales detection layers are used to improve the detection accuracy of small objects. In addition, this article proposes an optimized network pruning algorithm to solve the problem that the real-time performance of the algorithm cannot be satisfied due to the limited computing resources of the vehicle-mounted computing platform. The method of sparse scaling factor is used to improve the existing channel pruning algorithm. Compared to the YOLOv4, the YOLOV4-5D improves the mean average precision by 4.23% on the BDD data sets and 1.68% on the KITTI data sets. Finally, by pruning the model, the inference speed of YOLOV4-5D is increased 31.3% and the memory is only 98.1 MB when the detection accuracy is almost unchanged. Nevertheless, the proposed algorithm is capable of real-time detection at faster than 66 frames/s (fps) and shows higher accuracy than the previous approaches with a similar fps.

In recent years, considerable progress has been made in semantic segmentation of images with favorable environments. However, the environmental perception of autonomous driving under adverse weather conditions is still very challenging. In particular, the low visibility at nighttime greatly affects driving safety. In this paper, we aim to explore image segmentation in low-light scenarios, thereby expanding the application range of autonomous vehicles. The segmentation algorithms for road scenes based on deep learning are highly dependent on the volume of images with pixel-level annotations. Considering the scarcity of labeled large-scale nighttime data, we performed synthetic data collection and data style transfer using images acquired in daytime based on the autonomous driving simulation platform and generative adversarial network, respectively. In addition, we also proposed a novel nighttime segmentation framework (SFNET-N) to effectively recognize objects in dark environments, aiming at the boundary blurring caused by low semantic contrast in low-illumination images. Specifically, the framework comprises a light enhancement network which introduces semantic information for the first time and a segmentation network with strong feature extraction capability. Extensive experiments with Dark Zurich-test and Nighttime Driving-test datasets show the effectiveness of our method compared with existing state-of-the art approaches, with 56.9% and 57.4% mIoU (mean of category-wise intersection-over-union) respectively. Finally, we also performed real-vehicle verification of the proposed models in road scenes of Zhenjiang city with poor lighting. The datasets are available at https://github.com/pupu-chenyanyan/semantic-segmentation-on-nightime.

As self-driving vehicles become more prevalent, the speed and accuracy of detecting surrounding objects through onboard sensing technology have become increasingly important. The YOLOv8-QSD network is a novel anchor-free driving scene detection network that builds on YOLOv8 and ensures detection accuracy while maintaining efficiency. The network’s backbone employs structural reparameterization techniques to transform the Diverse Branch Block based model. To accurately detect small objects, it integrates features of different scales and implements a bifpn-based feature pyramid after the backbone. To address the challenge of long-range detection in driving scenarios, a query-based model with a new pipeline structure is introduced. The test results demonstrate that this algorithm outperforms YOLOv8 on the soda-a dataset in terms of both speed and accuracy. With an accuracy rate of 64.5% and reduced computational requirements of 7.1 GFLOPs, it satisfies the speed, precision, and cost-effectiveness requirements for commercial vehicles in high-speed road driving scenarios.

As a core function of autonomous driving (AD) and the Internet of Vehicles (IoV), accurately predicting the trajectory of vehicles can significantly improve traffic safety and reduce crash injuries. In this article, we propose an intention-aware non-autoregressive Transformer model with multi-attention learning for multimodal vehicle trajectory prediction. We first present social attention learning (SAL) where graph attention is properly integrated with the Transformer encoder so as to model the social interaction between vehicles. Then, the social and temporal dependency across consecutive frames is captured by temporal attention learning (TAL). The above social and temporal attention modules can be interleaved and stacked to achieve the coupled modeling and thus extract abundant features from trajectory data. To implement precise prediction as well as efficient inference, we further put forward an intention-aware decoder query generation approach to produce multiple possible trajectories concurrently. Finally, cross-attention learning (CAL) is devised to make full use of the encoded features, therefore, yielding future predictions. The proposed model is evaluated on two large-scale vehicle trajectory datasets and the experimental results verify that our algorithm achieves better performance compared with some state-of-the-art models. The root-mean-square error (RMSE) of the predicted trajectory over 5-s time horizon for the next generation simulation (NGSIM) and HighD datasets is 3.43 and 1.10 m, respectively.