In intelligent vehicles, it is essential to monitor the driver’s condition; however, recognizing the driver’s emotional state is one of the most challenging and important tasks. Most previous studies focused on facial expression recognition to monitor the driver’s emotional state. However, while driving, many factors are preventing the drivers from revealing the emotions on their faces. To address this problem, we propose a deep learning-based driver’s real emotion recognizer (DRER), which is a deep learning-based algorithm to recognize the drivers’ real emotions that cannot be completely identified based on their facial expressions. The proposed algorithm comprises of two models: (i) facial expression recognition model, which refers to the state-of-the-art convolutional neural network structure; and (ii) sensor fusion emotion recognition model, which fuses the recognized state of facial expressions with electrodermal activity, a bio-physiological signal representing electrical characteristics of the skin, in recognizing even the driver’s real emotional state. Hence, we categorized the driver’s emotion and conducted human-in-the-loop experiments to acquire the data. Experimental results show that the proposed fusing approach achieves 114% increase in accuracy compared to using only the facial expressions and 146% increase in accuracy compare to using only the electrodermal activity. In conclusion, our proposed method achieves 86.8% recognition accuracy in recognizing the driver’s induced emotion while driving situation.
Despite the advancement of advanced driver assistance systems (ADAS) and autonomous driving systems, surpassing the threshold of level 3 of driving automation remains a challenging task. Level 3 of driving automation requires assuming full responsibility for the vehicle’s actions, necessitating the acquisition of safer and more interpretable cues. To approach level 3, we propose a novel method for detecting driving vehicles and their brake light status, which is a crucial visual cue relied upon by human drivers. Our proposal consists of two main components. First, we introduce a fast and accurate one-stage brake light status detection network based on YOLOv8. Through transfer learning using a custom dataset, we enable YOLOv8 not only to detect the driving vehicle, but also to determine its brake light status. Furthermore, we present the publicly available custom dataset, which includes over 11,000 forward images along with manual annotations. We evaluate the performance of our proposed method in terms of detection accuracy and inference time on an edge device. The experimental results demonstrate high detection performance with an mAP50 (mean average precision at IoU threshold of ) ranging from to on the test dataset, along with a short inference time of ms on the Jetson Nano device. In conclusion, our proposed method achieves high accuracy and fast inference time in detecting brake light status. This contribution effectively improves safety, interpretability, and comfortability by providing valuable input information for ADAS and autonomous driving technologies.
A driver state estimation algorithm that uses multimodal vehicular and physiological sensor data is proposed. Deep learning is applied to the fused multimodal data rather than each modality being treated as a different feature. A convolutional neural network model is developed and the driver state estimation algorithm is implemented using Google TensorFlow. The results show that deep learning is a very promising approach for driver state estimation compared with previously studied algorithms, such as dynamic Bayesian networks.
As vehicles provide various services to drivers, research on driver emotion recognition has been expanding. However, current driver emotion datasets are limited by inconsistencies in collected data and inferred emotional state annotations by others. To overcome this limitation, we propose a data collection system that collects multimodal datasets during real-world driving. The proposed system includes a self-reportable HMI application into which a driver directly inputs their current emotion state. Data collection was completed without any accidents for over 122 h of real-world driving using the system, which also considers the minimization of behavioral and cognitive disturbances. To demonstrate the validity of our collected dataset, we also provide case studies for statistical analysis, driver face detection, and personalized driver emotion recognition. The proposed data collection system enables the construction of reliable large-scale datasets on real-world driving and facilitates research on driver emotion recognition. The proposed system is avaliable on GitHub.
The ever-increasing number of vehicles on the road puts pressure on car manufacturers to make their car fuel-efficient. With autonomous vehicles, we can find new strategies to optimize fuels. We propose a reinforcement learning algorithm that trains deep neural networks to generate a fuel-efficient velocity profile for autonomous vehicles given road altitude information for the planned trip. Using a highly accurate industry-accepted fuel economy simulation program, we train our deep neural network model. We developed a technique for adapting the heterogeneous simulation program on top of an open-source deep learning framework, and reduced dimension of the problem output with suitable parameterization to train the neural network much faster. The learned model combined with reinforcement learning-based strategy generation effectively generated the velocity profile so that autonomous vehicles can follow to control itself in a fuel efficient way. We evaluate our algorithm’s performance using the fuel economy simulation program for various altitude profiles. We also demonstrate that our method can teach neural networks to generate useful strategies to increase fuel economy even on unseen roads. Our method improved fuel economy by 8% compared to a simple grid search approach.
To meet the growing trend of stringent fuel economy regulations, automakers around the world are designing modules such as engines, motors, transmissions and batteries to be as efficient as possible. In order to verify the effect of these designs on the overall fuel efficiency of the vehicle, the vehicle equipped with each module is placed on the chassis dynamometer, driven to follow the target vehicle speed, and actual fuel efficiency is measured. These tests are traditionally performed by human operators, but are now being replaced by robots (physical or software) to ensure the accuracy and reliability of test results. Although the conventionally proposed proportional integral (PI)-based controller has a simple structure and is easy to implement, it requires the process of finding the optimal gain whenever the test conditions such as vehicle or drive cycle change, which is difficult and time consuming. In this study, we propose a proportional integral controller gain adjustment algorithm using deep reinforcement learning. The reinforcement learning agent learns to dynamically modify the PI gain value of the acceleration/deceleration pedal to better follow the target vehicle in a simulation environment. The perturbation is used in each training episode to reduce the difference between the simulation and real testing environment. Upon completion of the training process, the trained agent performs an adjustment process that generates a reference gain table. We then use this reference gain table to perform a real test. The performance of the proposed system was evaluated using Hyundai Tucson HEV (NX4) on an AVL chassis dynamometer. We also compared the performance of our proposed algorithm to traditional fuzzy logic-based PI controllers. The obtained experimental results show that the proposed control system achieved a performance improvement of aounrd 46.8% compared to the conventional PI control system in terms of root mean square error.