19 February 2021 in IEEE Sensors Journal
In EEG based driving fatigue detection, the redundant EEG channels would increase the probability of introducing noise and heaving calculation burden. Aiming at this point, we built up an EEG dataset with thirty electrodes from every brain lobe, proposed a driving fatigue detection approach based on brain functional network. And provide a possible way for the key EEG electrode and rhythm investigation with these brain networks. First, a simulation driving experiment was set up to obtain EEG of multiple subjects during a long-term monotonous cognitive task. The feasibility of the collected data was verified with PSD. For brain functional network construction, partial directed coherence (PDC) is used to calculate the correlation between EEG channels. Then, four graph related features (i.e., node degree, clustering coefficient, characteristic path length and local efficiency) are extracted for fatigue detection with SVM as classifier. After that, we get the key electrodes with the feature of betweenness centrality. The results show that the low frequency rhythms are more distinguishable than that of high frequency. The best classification accuracy of 87.16% is achieved in delta, which is 18.9% higher than that of the baseline method with PSD. At last, we verified the effectiveness of the selected key electrodes, and drew the conclusion that the key EEG rhythm corresponding to different features may not be consistent.
13 July 2022 in IEEE Transactions on Circuits and Systems for Video Technology
As an essential nonverbal cue, the human gaze reveals human intentions and plays a crucial role in human daily activities. Therefore, automatic detection of the person’s gaze target has drawn the interests of the computer vision community. This is useful not only for identifying whether children are attentive in class but also for locating items of interest to humans in retail settings. Existing gaze-following methods have only explored and exploited the scenes context and the head cues. Considering the significance of human-object interaction in understanding human intentions, we present the Visual-Spatial Graph and introduce a graph attention network to analyze the interaction probability between the human and elements in the scene. Then the interaction probability inferred from the visual-spatial information that is aggregated by the attention mechanism can be transformed into an interactive attention map that depicts the areas people care about. In addition, we construct a transformer as an encoder to integrate the features extracted by the scene and head pathways aiming to decode the gaze target. After introducing interactive attention, our proposed method achieves outstanding performance on two benchmarks: GazeFollow and VideoAttentionTarget. Our code is available at https://github.com/nkuhzx/VSG-IA.
20 May 2022 in IEEE Robotics and Automation Letters
Human trajectory prediction is a crucial yet challenging problem, which is of fundamental importance to robotics and autonomous driving vehicles. The core challenge lies in effectively modeling the socially aware spatial interaction and complex temporal dependencies among crowds. However, previous methods usually model spatial and temporal information separately and only use individual features, while ignoring group-level motion characteristics. We propose a novel trajectory prediction framework termed GA-STT, a group aware spatial-temporal transformer network to address these issues. Specifically, we first get the individual representations supervised by group-based annotations. Then, we model the complex spatial-temporal interactions with spatial and temporal transformers separately and fuse spatial-temporal embedding through the cross-attention mechanism. Results on the publicly available ETH/UCY datasets show that our model outperforms the state-of-the-art method by 19.4% in average displacement error(ADE) and 16.9% in final displacement error(FDE) and successfully predicts complex spatial-temporal interactions.
23 March 2022 in IEEE Transactions on Instrumentation and Measurement
Inferring gaze target or gaze following is an effective way to understand human actions and intentions, which makes quite a challenge. Some existing studies on gaze estimation cannot accurately locate the gaze target in a 3D scene by gaze direction alone, while other studies on gaze following have failed to exploit the contexts in the 3D scene. In this article, we make full use of the information obtained by the RGB-D camera and innovatively expand the gaze target estimation from 2D image to 3D space through the predicted 3D gaze vector. Specifically, we rebuild a new 3D gaze-following dataset, RGB-D Attention dataset, which contains 3D real-world gaze behaviors. In addition, we extend the depth information for the GazeFollow dataset to utilize its diverse scene information in the training process of 3D gaze following. Then, considering the gaze direction as a crucial clue, we propose a novel gaze vector space containing 3D information and a 3D gaze pathway for learning the gaze behavior in the 3D scene. After two-stage training, the entire model can output the predicted 3D gaze vector and the predicted gaze heatmap, which are used to estimate the 3D gaze target in the inference algorithm. Experiments in the 3D scenes show that our method can reduce the predicted average distance error to 0.307 m and the predicted average angle error to 19.8°. Compared with the state-of-the-art gaze inference method, our proposed method has reduced the prediction error by more than 45%. Our web page is at https://sites.google.com/view/3dgazefollow .
26 August 2022 in IEEE Transactions on Neural Networks and Learning Systems
For a better intention inference, we often try to figure out the emotional states of other people in social communications. Many studies on affective computing have been carried out to infer emotions through perceiving human states, i.e., facial expression and body posture. Such methods are skillful in a controlled environment. However, it often leads to misestimation due to the deficiency of effective inputs in unconstrained circumstances, that is, where context-aware emotion recognition appeared. We take inspiration from the advanced reasoning pattern of humans in perceived emotion recognition and propose the hierarchical context-based emotion recognition method with scene graphs. We propose to extract three contexts from the image, i.e., the entity context, the global context, and the scene context. The scene context contains abstract information about entity labels and their relationships. It is similar to the information processing of the human visual sensing mechanism. After that, these contexts are further fused to perform emotion recognition. We carried out a bunch of experiments on the widely used context-aware emotion datasets, i.e., CAER-S, EMOTIC, and BOdy Language Dataset (BoLD). We demonstrate that the hierarchical contexts can benefit emotion recognition by improving the accuracy of the SOTA score from 84.82% to 90.83% on CAER-S. The ablation experiments show that hierarchical contexts provide complementary information. Our method improves the F1 score of the SOTA result from 29.33% to 30.24% (C-F1) on EMOTIC. We also build the image-based emotion recognition task with BoLD-Img from BoLD and obtain a better emotion recognition score (ERS) score of 0.2153.
5 December 2023 in Information Fusion