Driver attention prediction can solve the problem of ‘Where should the driver pay attention?’, Most previous methods are designed to predict regional attention with redundant regions. Furthermore, popular spatial-temporal feature extraction networks such as ConvLSTM and 3D-CNN are difficult to achieve real-time. To overcome these difficulties, we propose an Adaptive Short-temporal Induced Aware Fusion Network (ASIAF-Net) for region-level and object-level driver attention prediction. 1 In ASIAF-Net, we design an Attention Related Spatial Feature Encoder (AF-Encoder) and an Induced Aware Fusion Network (IAF-Net) as the main network; with an Association Analysis Cell (AAC), the AF-Encoder makes it possible to effectively capture the relationship information of different objects. Considering most vital visual cues from moving objects, we propose a Self-adaptive Short-temporal Feature Extraction Module (SSFE-Module) to obtain inter-frame motion features. In IAF-Net, a Multi-scale Driver Attention Region Prediction Branch is designed to predict the regional attention, and an Object Saliency Estimation Branch is proposed to fuse the perception results and the regional attention map to estimate the object-level attention. Experiments show that the proposed ASIAF-Net can predict driver’s attention on regions and objects more robustly and precisely than state-of-the-art methods on three datasets, and that it achieves real-time on our ADAS platform.

On-board vision based traffic risk assessment is a challenging task for intelligent driving systems, which has some special challenges including spatial-temporal feature extraction, different judgements of risks, real-time requirement, lacking data, etc. To overcome these difficulties, we propose a novel Temporal Shift and Spatial Attention based Two-stream Network (TSSAT-Net) for on-board vision based traffic risk assessment. Firstly, we build new judgement measures that integrate actual driving experience and scenario complexity, and release an on-board vision based traffic risk assessment dataset. Secondly, a novel weighted Temporal Shift Module (weighted-TSM) based two-stream network is proposed; unlike previous methods that rely on complex and time-consuming 3D CNN or LSTM calculation, the proposed two-stream network can effectively extract spatial-temporal features using a weighted temporal shift mechanism with just 2D CNN calculation requirement. Thirdly, a spatial and channel attention mechanism is proposed to make the TSSAT-Net more focus on the features closely related to traffic risks, avoiding redundant information in complex traffic scenarios. Experiments based on the released comprehensive dataset show that our method achieves the state-of-the-art classification accuracy in real-time.

You Only Look Once (YOLO) deep network can detect objects quickly with high precision and has been successfully applied in many detection problems. The main shortcoming of YOLO network is that YOLO network usually cannot achieve high precision when dealing with small-size object detection in high resolution images. To overcome this problem, we propose an effective region proposal extraction method for YOLO network to constitute an entire detection structure named ACF-PR-YOLO, and take the cyclist detection problem to show our methods. Instead of directly using the generated region proposals for classification or regression like most region proposal methods do, we generate large-size potential regions containing objects for the following deep network. The proposed ACF-PR-YOLO structure includes three main parts. Firstly, a region proposal extraction method based on aggregated channel feature (ACF) is proposed, called ACF based region proposal (ACF-PR) method. In ACF-PR, ACF is firstly utilized to fast extract candidates and then a bounding boxes merging and extending method is designed to merge the bounding boxes into correct region proposals for the following YOLO net. Secondly, we design suitable YOLO net for fine detection in the region proposals generated by ACF-PR. Lastly, we design a post-processing step, in which the results of YOLO net are mapped into the original image outputting the detection and localization results. Experiments performed on the Tsinghua-Daimler Cyclist Benchmark with high resolution images and complex scenes show that the proposed method outperforms the other tested representative detection methods in average precision, and that it outperforms YOLOv3 by $13.69\%$ average precision and outperforms SSD by $25.27\%$ average precision.

Machine vision based vehicle counting and traffic flow estimation are challenging problems especially for dense traffic scenarios. Previous line of interest (LOI) counting methods rarely focus on dense scenarios and their performance largely relies on the accuracy of tracking. Avoiding the use of complex tracking methods, an LOI counting framework is proposed to address the bi-directional LOI counting problem in dense scenarios. There are three main contributions. Firstly, instead of treating the LOI vehicle counting problem as a combination of detecting and tracking of individual vehicles, the bi-directional traffic flow is taken as a whole and a novel spatio-temporal counting feature (STCF) is proposed for extracting bi-directional traffic flow features in dense traffic scenarios. Secondly, without relying on a multi-target tracking process for tracking and counting each vehicle, a counting network is proposed, called the counting Long Short-Term Memory (cLSTM) network, to do analysis of the bi-directional STCF features and vehicle counting in successive video frames. Lastly, an estimation model is designed for estimating traffic flow parameters including speed, volume and density. Experiments performed on the UA-DETRAC dataset and the captured videos show that the proposed vehicle counting method outperforms the tested representative LOI counting methods in both accuracy and speed, and that the proposed framework can efficiently estimate traffic flow parameters including speed, volume and density in real time.

Vehicle counting and traffic volume estimation on traffic videos has gained extensive attention from multimedia and computer vision communities. Recent vehicle counting and volume estimation methods, including detection based and time-spatial image (TSI) based methods have achieved significant improvements. However, how to balance the accuracy and speed is still a challenge to this task. In this paper, we design a fast and accurate vehicle counting and traffic volume estimation method. Firstly, traffic videos are converted to TSIs and we annotate the vehicle locations in TSIs manually. Then, we design a simple TSI density map estimation network which utilizes attention mechanism to strengthen the features in the traffic locations for vehicle counting. Finally, we use the parameters obtained from the vehicle counting network to further estimate the traffic volume. Experiments on UA-DETRAC dataset demonstrate that the vehicle counting network not only takes a balance between counting accuracy and speed, but also well estimates the traffic volume when the video data is insufficient.

Describing a traffic scenario from the driver’s perspective is a challenging process for Advanced Driving Assistance System (ADAS), involving different sub-tasks of detection, tracking, segmentation, etc. Previous methods mainly focus on independent sub-tasks and have difficulties to comprehensively describe the incidents. In this study, this problem is novelly treated as a video captioning task, and a Guidance Attention Captioning Network (GAC-Network) structure is proposed for describing the incidents in a concise single sentence. In GAC-Network, an Attention based Encoder-Decoder Net (AED-Net) is built as the main network; with the temporal spatial attention mechanisms, the AED-Net make it possible to effectively reject the unimportant traffic behaviors and redundant backgrounds. Considering various driving scenarios, the Spatio-Temporal Layer Normalization is used to improve the generalization ability. To generate captions for incidents in driving, the novel Guidance Module is proposed to boost the encoder-decoder model to generate words in a caption, which have better relationship to the past and future words. Because there is no public dataset for captioning of driving scenarios, the Traffic Video Captioning (TVC) dataset is released for the video captioning task in driving scenarios. Experimental results show that the proposed methods can fulfill the captioning task for complex driving scenarios, and achieve higher performance than the methods for comparison, including at least 2.5%, 1.8%, 3.6%, and 13.1% better results on BLEU_1, METEOR, ROUGE_L and CIDEr, respectively.