With the development of intelligent automotive human-machine systems, driver emotion detection and recognition has become an emerging research topic. Facial expression-based emotion recognition approaches have achieved outstanding results on laboratory-controlled data. However, these studies cannot represent the environment of real driving situations. In order to address this, this paper proposes a facial expression-based on-road driver emotion recognition network called FERDERnet. This method divides the on-road driver facial expression recognition task into three modules: a face detection module that detects the driver’s face, an augmentation-based resampling module that performs data augmentation and resampling, and an emotion recognition module that adopts a deep convolutional neural network pre-trained on FER and CK+ datasets and then fine-tuned as a backbone for driver emotion recognition. This method adopts five different backbone networks as well as an ensemble method. Furthermore, to evaluate the proposed method, this paper collected an on-road driver facial expression dataset, which contains various road scenarios and the corresponding driver’s facial expression during the driving task. Experiments were performed on the on-road driver facial expression dataset that this paper collected. Based on efficiency and accuracy, the proposed FERDERnet with Xception backbone was effective in identifying on-road driver facial expressions and obtained superior performance compared to the baseline networks and some state-of-the-art networks.
30 November 2021 in IEEE Transactions on Computational Social Systems
Driver’s emotion recognition is vital to improving driving safety, comfort, and acceptance of intelligent vehicles. This article presents a cognitive-feature-augmented driver emotion detection method that is based on emotional cognitive process theory and deep networks. Different from the traditional methods, both the driver’s facial expression and cognitive process characteristics (age, gender, and driving age) were used as the inputs of the proposed model. Convolutional techniques were adopted to construct the model for driver’s emotion detection simultaneously considering the driver’s facial expression and cognitive process characteristics. A driver’s emotion data collection was carried out to validate the performance of the proposed method. The collected dataset consists of 40 drivers’ frontal facial videos, their cognitive process characteristics, and self-reported assessments of driver emotions. Another two deep networks were also used to compare recognition performance. The results prove that the proposed method can achieve well detection results for different databases on the discrete emotion model and dimensional emotion model, respectively.
3 March 2021 in IEEE Transactions on Affective Computing
In this paper, a new dataset, the driver emotion facial expression (DEFE) dataset for drivers spontaneous emotions analysis is introduced. The dataset includes facial expression recordings from 60 participants during driving. After watching a selected video-audio clip to elicit a specic emotion, each participant completed the driving tasks in the same driving scenario and rated his/her emotional responses during the driving processes from the aspects of dimensional emotion method and discrete emotion method. The study also conducted classication experiments to recognize the scales of arousal, valence, dominance, as well as the emotion category and intensity to establish baseline results for the proposed dataset. Furthermore, this paper compared emotion recognition results difference through facial expressions between dynamic driving and static life scenarios. The results showed signicant differences in the AUs presence of facial expressions between dynamic driving and static life scenarios, Therefore, to accurately recognize the drivers emotions to establish a reliable emotion-aware human-machine interaction system, thereby improving driving safety and comfort, publishing a human emotion dataset specically for the driver is necessary. The proposed dataset will be publicly available so that researchers worldwide can use it to develop and examine their driver emotion analysis methods.
Human emotions are integral to daily tasks, and driving is now a typical daily task. Creating a multi-modal human emotion dataset in driving tasks is an essential step in human emotion studies. we conducted three experiments to collect multimodal psychological, physiological and behavioural dataset for human emotions (PPB-Emo). In Experiment I, 27 participants were recruited, the in-depth interview method was employed to explore the driver’s viewpoints on driving scenarios that induce different emotions. For Experiment II, 409 participants were recruited, a questionnaire survey was conducted to obtain driving scenarios information that induces human drivers to produce specific emotions, and the results were used as the basis for selecting video-audio stimulus materials. In Experiment III, 40 participants were recruited, and the psychological data and physiological data, as well as their behavioural data were collected of all participants in 280 times driving tasks. The PPB-Emo dataset will largely support the analysis of human emotion in driving tasks. Moreover, The PPB-Emo dataset will also benefit human emotion research in other daily tasks.
6 December 2023 in IEEE Transactions on Intelligent Vehicles
As a new round of technological revolution and industrial transformation accelerates, intelligent connected vehicles have emerged as carriers for advanced technology integration across industries and multiple fields. The intelligent cockpit serves as a comprehensive application space for various new technologies of intelligent connected vehicles, fostering integration among intelligent technology, products, and markets to form a virtuous circle. Developing intelligent cockpits helps to enhance the overall competitiveness and innovation capabilities of the industry, driving the growth of the intelligent automotive industry ecosystem. This paper proposes the definition of the intelligent cockpit for automobiles, classifying it into five levels from “functional cockpit” to “fully cognitive intelligent cockpit” based on the human-machine-environment perspective. The paper also provides a preliminary description of the “3-horizontal-3- vertical” technical architecture of the intelligent cockpit and outlines the development roadmap. Additionally, the paper proposes a comprehensive evaluation index system and method for assessing the intelligence of automotive cockpits. By addressing these key aspects, we hope to shed light on the potential of intelligent cockpits in advancing the field of intelligent connected vehicles.
26 December 2022 in IEEE Transactions on Systems, Man, and Cybernetics: Systems
Advances in technologies, such as intelligent connected vehicles and the metaverse are driving the rapid development of automotive intelligent cockpits. From the perspective of the cyber–physical–social system (CPSS), this study proposed the intelligent cockpit composition framework which includes three layers of perception, cognition and decision, and interaction. Meanwhile, we also describe the relationship between the intelligent cockpit framework and the outside environment. The framework can dynamically perceive and understand humans, and provide feedback on the understanding results, which is beneficial to provide a safe, efficient, and enjoyable experience for humans in the intelligent cockpit. In the cognition and decision layers of the proposed framework, we design a case study of active empathetic auditory regulation of driver anger, focusing on improving road traffic safety. We conducted an in-depth interview experiment and designed two auditory regulation materials of active empathy speech and text-to-speech (TTS) speech. Next, 30 participants were recruited, and they completed a total of 240 anger-regulated driving experiments in the straight and obstacle avoidance scenarios. Finally, we quantitatively analyzed and compared the participants’ subjective feelings, physiological changes, driving behaviors, and driving risks, as well as validated the driver anger regulation quality of AES and TTS. The proposed research methods results are beneficial to the design of future intelligent cockpit emotion regulation systems, toward a better intelligent cockpit.