Facial expression recognition (FER) is a significant task for machines to understand the emotional changes in human beings. However, accurate hand-crafted features that are highly related to changes in expression are difficult to extract because of the influences of individual difference and variations in emotional intensity. Therefore, features that can accurately describe the changes in facial expressions are urgently required. Method: A weighted mixture deep neural network (WMDNN) is proposed to automatically extract features that are effective for FER tasks. Several pre-processing approaches, such as face detection, rotation rectification, and data augmentation, are implemented to restrict regions for FER. Two channels of facial images, including facial grayscale images and their corresponding local binary pattern (LBP) facial images, are processed by WMDNN. Expression-related features of facial grayscale images are extracted by fine-tuning a partial VGG16 network, the parameters of which are initialized using VGG16 model trained on ImageNet database. Features of LBP facial images are extracted by a shallow convolutional neural network (CNN) built based on DeepID. The outputs of both channels are fused in a weighted manner. The result of final recognition is calculated using softmax classification. Results: Experimental results indicate that the proposed algorithm can recognize six basic facial expressions (happiness, sadness, anger, disgust, fear, and surprise) with high accuracy. The average recognition accuracies for benchmarking datasets “CK+,” “JAFFE,” and “Oulu-CASIA” are 0.970, 0.922, and 0.923, respectively. Conclusions: The proposed FER method outperforms state-of-the-art FER methods based on hand-crafted features or deep networks using one channel. Compared with deep networks that use multiple channels, our proposed network can achieve comparable performance with easier procedures. Fine-tuning is effective to FER tasks with a well pre-trained model if sufficient samples cannot be collected.

It is critical for autonomous vehicles to accurately forecast the future trajectories of surrounding agents to avoid collisions. However, capturing the complex interactions between agents in complex urban scenes is challenging. As a result, complex interactions may impair trajectory prediction accuracy. A trajectory prediction network with an enhanced Graph Transformer (TP-EGT) is proposed to forecast the future trajectories of traffic-agents. A collision-aware Graph Transformer is introduced to capture the complex social interactions between traffic-agents. Following that, an additional interaction prediction task that could predict the interaction probabilities between agents is proposed to mitigate the over-smoothing issue of the Graph Transformer via a multi-task learning strategy. Afterward, the trajectory prediction performance is improved with additional interaction probabilities, which are beneficial for the decision-making and planning modules of autonomous vehicles. Quantitative and qualitative evaluations of TP-EGT on the ETH/UCY and ApolloScape databases demonstrate that the trajectory prediction accuracy of TP-EGT is comparable to the state-of-the-art baseline methods, and the predicted interaction probabilities can help autonomous vehicles comprehend the complex traffic scenes.

The booming self-driving cars need to understand the behaviors of other road users for better performance. Recognizing pedestrians’ crossing intentions is one of the most critical capabilities of self-driving cars to guarantee safe operations in the urban environment. Some researchers try to predict the future trajectories of pedestrians to avoid a potential collision. Others extract skeleton features from pedestrians to detect specific actions related to the crossing intention. All these methods either neglect the abundant appearance information of pedestrians, or work poorly under severe conditions, e.g., pedestrians standing far away, dim light, and occlusions. In this work, a pedestrian crossing intention recognition (PCIR) framework is proposed to recognize pedestrians’ crossing intention. A module for searching targets of interest is introduced to find the pedestrians who are likely to cross the streets, and perform scene perception simultaneously. An action recognition module utilizes a 3D convolutional neural network (CNN) to automatically extract spatiotemporal features that imply early actions before crossing, like limb movements. A distance encoding module makes full use of the contextual cues, e.g., distances between pedestrians and the oncoming vehicle, local traffic scenes around pedestrians, and vehicle speeds, to improve the recognition accuracy obtained from the action recognition module only. Finally, the PCIR framework fuses these factors to predict pedestrians’ crossing intentions by minimizing a focal loss. Experimental evaluations verify the effectiveness of the proposed method in recognizing pedestrians’ crossing intentions. Comparisons with the skeleton-based methods reveal the robustness of the PCIR framework in the urban environment.

Pedestrian trajectory prediction in dynamic scenes remains a challenging and critical problem in numerous applications, such as self-driving cars and socially aware robots. Challenges concentrate on capturing pedestrians' motion patterns and social interactions, as well as handling the future uncertainties. Recent studies focus on modeling pedestrians' motion patterns with recurrent neural networks, capturing social interactions with pooling- or graph-based methods, and handling future uncertainties by using the random Gaussian noise as the latent variable. However, they do not integrate specific obstacle avoidance experiences (OAEs) that may improve prediction performance. For example, pedestrians' future trajectories are always influenced by others in front. Here, we propose the Graph-based Trajectory Predictor with Pseudo-Oracle (GTPPO), an encoder-decoder-based method conditioned on pedestrians' future behaviors. Pedestrians' motion patterns are encoded with a long short-term memory unit, which introduces temporal attention to highlight specific time steps. Their interactions are captured by a graph-based attention mechanism, which draws OAE into the data-driven learning process of graph attention. Future uncertainties are handled by generating multimodal outputs with an informative latent variable. Such a variable is generated by a novel pseudo-oracle predictor, which minimizes the knowledge gap between historical and ground-truth trajectories. Finally, the GTPPO is evaluated on ETH, UCY, and Stanford Drone datasets, and the results demonstrate state-of-the-art performance. Besides, the qualitative evaluations show successful cases of handling sudden motion changes in the future. Such findings indicate that GTPPO can peek into the future.