Multi-modal fusion is a basic task of autonomous driving system perception, which has attracted many scholars' attention in recent years. The current multi-modal fusion methods mainly focus on camera data and LiDAR data, but pay little attention to the kinematic information provided by the sensors of the vehicle, such as acceleration, vehicle speed, angle of rotation. These information are not affected by complex external scenes, so it is more robust and reliable. In this paper, we introduce the existing application fields of vehicle information and the research progress of related methods, as well as the multi-modal fusion methods based on information. We also introduced the relevant information of the vehicle information dataset in detail to facilitate the research as soon as possible. In addition, new future ideas of multi-modal fusion technology for autonomous driving tasks are proposed to promote the further utilization of vehicle information.

Multi-modal sensor fusion techniques have promoted the development of autonomous driving, while perception in the complex environment remains a challenging problem. In order to tackle the problem, we propose the Open Multi-modal Perception dataset (OpenMPD), a multi-modal perception benchmark objected at difficult examples. Compared with existing datasets, OpenMPD focuses more on those complex traffic scenes in urban areas with overexposure or darkness, crowded environment, unstructured roads and intersections. It acquires the multi-modal data through a vehicle with six cameras and four LiDAR for a 360-degree field of view and collected 180 clips of 20-second synchronized images at 20 Hz and point clouds at 10 Hz. Particularly, we applied a 128-beam LiDAR to provide Hi-Res point clouds to better understand the 3D environment and sensor fusion. We sampled 15 K keyframes at equal intervals from clips for annotations, including 2D/3D object detections, 3D object tracking, and 2D semantic segmentation. Moreover, we provide four benchmarks for all tasks to evaluate algorithms and conduct extensive experiments of 2D/3D detection and segmentation on OpenMPD. Data and further information are available at http://www.openmpd.com/ .

Driving behavior classification is an important direction in the field of social transportation systems and advanced driving assistance system (ADAS), which has attracted more and more attention in recent years. An accurate driving behavior classification algorithm plays a great role in traffic safety, energy saving, and other fields. In this article, we propose a novel vehicle speed and image fusion for driving behavior classification network (SIFDriveNet), which classifies driver behaviors into normal driving, aggressive driving, and drowsy driving. Our method has the following key advantages. First, in the research of driving behavior classification, we are the first to introduce a 2-D image with rich roadside information and convert speeds into a 2-D spectrogram expressing the time–frequency characteristics of speeds through short-time Fourier transform (STFT) while unifying the data space of image information and speed information. Second, we propose a tensor fusion method based on weight decomposition to fully fuse the vectors of the two modalities. This method maps the tensor outer product results to the low-dimensional space through weight decomposition and has a low computational cost while maintaining the fusion effect of the tensor outer product. In addition, we evaluated our model on the public UAH-DriveSet and compared it with the most advanced model. Experimental results show that our model has a better performance, and F1-score is 97.9% on all roads. Especially on the secondary road, our F1-score is 99.4%. Also, our model has strong generalization, and we have reached 99.3% F1 in distracted driving multimodal dataset. In addition, the inference speed reaches 411 FPS, enabling real-time needs. The code is available on https://github.com/alu222/SIFDriveNet .

Vehicle re-identification is the task of identifying the same vehicle in different environments and from different angles and cameras. It is more challenging than re-identification of humans: 1)small differences between vehicles of the same model make it difficult to capture their subtle characteristics; 2)vehicles of different types and colors may have similar characteristics from different viewpoints or external conditions. To address these challenges, we propose a TVG-ReID network, using a Transformer network to enhance features extracted from a CNN backbone network. A vehicle knowledge graph transfer method(Vehicle-Graph) is proposed, which treats each vehicle as a node in a graph, where simple information is transmitted through edges to constrain the distance of the nodes in a metric learning manner. Experiments on two vehicle re-identification datasets demonstrate the good performance of our proposed model.

Collaborative perception has attracted growing interest from academia and industry due to its potential to enhance perception accuracy, safety, and robustness in autonomous driving through multi-agent information fusion. With the advancement of Vehicle-to-Everything (V2X) communication, numerous collaborative perception datasets have emerged, varying in cooperation paradigms, sensor configurations, data sources, and application scenarios. However, the absence of systematic summarization and comparative analysis hinders effective resource utilization and standardization of model evaluation. As the first comprehensive review focused on collaborative perception datasets, this work reviews and compares existing resources from a multi-dimensional perspective. We categorize datasets based on cooperation paradigms, examine their data sources and scenarios, and analyze sensor modalities and supported tasks. A detailed comparative analysis is conducted across multiple dimensions. We also outline key challenges and future directions, including dataset scalability, diversity, domain adaptation, standardization, privacy, and the integration of large language models. To support ongoing research, we provide a continuously updated online repository of collaborative perception datasets and related literature: https://github.com/frankwnb/Collaborative-Perception-Datasets-for-Autonomous-Driving.