3D Multi-object tracking (MOT) ensures consistency during continuous dynamic detection, conducive to subsequent motion planning and navigation tasks in autonomous driving. However, camera-based methods suffer in the case of occlusions and it can be challenging to track the irregular motion of objects for LiDAR-based methods accurately. Some fusion methods work well but do not consider the untrustworthy issue of appearance features under occlusion. At the same time, the false detection problem also significantly affects tracking. As such, we propose a novel camera-LiDAR fusion 3D MOT framework based on Combined Appearance-Motion Optimization (CAMO-MOT), which uses both camera and LiDAR data and significantly reduces tracking failures caused by occlusion and false detection. For occlusion problems, we are the first to propose an occlusion head to select the best object appearance features multiple times effectively, reducing the influence of occlusions. To decrease the impact of false detection in tracking, we design a motion cost matrix based on confidence scores which improve the positioning and object prediction accuracy in 3D space. As existing multi-object tracking methods always evaluate each category separately and do not consider the mismatch between objects of different categories, we also propose to build a multi-category cost to implement multi-object tracking in multi-category scenes. A series of validation experiments are conducted on the KITTI and nuScenes tracking benchmarks. Our proposed method achieves state-of-the-art performance with 79.99% HOTA and the lowest identity switches (IDS) value (23 for Car and 137 for Pedestrian) among all multi-modal MOT methods on the KITTI test dataset. And our method achieves state-of-the-art performance among all algorithms on the nuScenes test dataset with 75.3% AMOTA.

Multi-modal fusion is a basic task of autonomous driving system perception, which has attracted many scholars' attention in recent years. The current multi-modal fusion methods mainly focus on camera data and LiDAR data, but pay little attention to the kinematic information provided by the sensors of the vehicle, such as acceleration, vehicle speed, angle of rotation. These information are not affected by complex external scenes, so it is more robust and reliable. In this paper, we introduce the existing application fields of vehicle information and the research progress of related methods, as well as the multi-modal fusion methods based on information. We also introduced the relevant information of the vehicle information dataset in detail to facilitate the research as soon as possible. In addition, new future ideas of multi-modal fusion technology for autonomous driving tasks are proposed to promote the further utilization of vehicle information.

Driving behavior classification is an important direction in the field of social transportation systems and advanced driving assistance system (ADAS), which has attracted more and more attention in recent years. An accurate driving behavior classification algorithm plays a great role in traffic safety, energy saving, and other fields. In this article, we propose a novel vehicle speed and image fusion for driving behavior classification network (SIFDriveNet), which classifies driver behaviors into normal driving, aggressive driving, and drowsy driving. Our method has the following key advantages. First, in the research of driving behavior classification, we are the first to introduce a 2-D image with rich roadside information and convert speeds into a 2-D spectrogram expressing the time–frequency characteristics of speeds through short-time Fourier transform (STFT) while unifying the data space of image information and speed information. Second, we propose a tensor fusion method based on weight decomposition to fully fuse the vectors of the two modalities. This method maps the tensor outer product results to the low-dimensional space through weight decomposition and has a low computational cost while maintaining the fusion effect of the tensor outer product. In addition, we evaluated our model on the public UAH-DriveSet and compared it with the most advanced model. Experimental results show that our model has a better performance, and F1-score is 97.9% on all roads. Especially on the secondary road, our F1-score is 99.4%. Also, our model has strong generalization, and we have reached 99.3% F1 in distracted driving multimodal dataset. In addition, the inference speed reaches 411 FPS, enabling real-time needs. The code is available on https://github.com/alu222/SIFDriveNet .

Driving behavior classification is a critical component of social transportation systems and advanced driver assistance systems, and it has gained increasing attention in recent years. Accurate classification algorithms for driving behavior play a significant role in enhancing traffic safety, energy conservation, and related fields. In this article, we propose a novel driving behavior classification network named feature-attention- embedding-based multimodal-fusion driving-behavior-classification network (FMDNet). FMDNet incorporates eight types of data, including acceleration along the x-axis, y-axis, z-axis, roll angle, pitch angle, yaw angle, roadside image, and vehicle speed, to classify driving behavior. To effectively fuse features extracted from different modalities, taking into account their varying importance, we introduce the feature attention embedding-based fusion module (FAEF) as our fusion strategy. This fusion strategy enhances the network’s capability to capture meaningful features by incorporating two feature attention embedding units that delve deeper into the interplay between different modes. Furthermore, we provide further validation of the effectiveness of our approach through extensive ablation experiments to investigate and analyze the impact of various modal data on the classification of driving behavior. Our proposed FMDNet achieves state-of-the-art performance on the public UAH-DriveSet dataset, demonstrating its effectiveness with an impressive F1-score of 99.0%. Additionally, the robustness of our model is confirmed on distracted dataset, achieving a remarkable F1-score of 99.7%. The model’s outstanding performance on both the UAH-DriveSet dataset and the distracted-dataset highlights its capabilities and potential for real-world applications.

Vehicle re-identification is the task of identifying the same vehicle in different environments and from different angles and cameras. It is more challenging than re-identification of humans: 1)small differences between vehicles of the same model make it difficult to capture their subtle characteristics; 2)vehicles of different types and colors may have similar characteristics from different viewpoints or external conditions. To address these challenges, we propose a TVG-ReID network, using a Transformer network to enhance features extracted from a CNN backbone network. A vehicle knowledge graph transfer method(Vehicle-Graph) is proposed, which treats each vehicle as a node in a graph, where simple information is transmitted through edges to constrain the distance of the nodes in a metric learning manner. Experiments on two vehicle re-identification datasets demonstrate the good performance of our proposed model.

Lane detection is a critical part of autonomous driving technology. When difficult situations are encountered (i.e., adverse light, severe occlusion), the lane detection task is still challenging. However, previous methods strongly depend on the extracted image features and ignore other features. It is necessary to consider the information from other modalities to assist the model for lane detection, especially in the task of curved lane detection. In this paper, considering that the vehicle steering angle is closely related to the visual feature of lane lines, we propose a novel model named Image-Angle Fusion Network (IAFNet) to solve the lane detection problem by fusing vehicle steering angle features with image features. To make the steering angle features better match the image features, we use the tensor outer product to extend the dimensionality of the steering angle information. A lightweight Image-Angle cross-attention module (LIA-CAM) is proposed to learn the implicit relationship between steering angles and visual features of lane lines, aimed at improving the performance of our model in difficult situations. To guide the network to retain the correct steering angle information, we introduced regression prediction loss of steering angle. Besides, we also released a new dataset based on the Udacity dataset: ImageAngle-Udacity (IA-Udacity) dataset. Extensive experiments on the IA-Udacity dataset show that our method outperforms the current state-of-the-art methods showing both higher efficiency and accuracy. Code and data are available on https://github.com/gongyan1/LIA-CAM.