Given an arbitrary speech clip or text information as input, the proposed work aims to generate a talking face video with accurate lip synchronization. Existing works mainly have three limitations. (1) A single-modal learning is adopted with either audio or text as input, hence it lacks the complementarity of multimodal inputs. (2) Each frame is generated independently, hence it ignores the temporal dependency between consecutive frames. (3) Each face image is generated by the traditional convolution neural network (CNN) with a local receptive field, hence it cannot effectively capture the spatial dependency within internal representations of face images. To overcome these problems above, we decompose the talking face generation task into two steps: mouth landmarks prediction and video synthesis. First, a multimodal learning method is proposed to generate accurate mouth landmarks with multimedia inputs ( both text and audio). Second, a network named Face2Vid is proposed to generate video frames conditioned on the predicted mouth landmarks. In Face2Vid, the optical flow is employed to model the temporal dependency between frames, meanwhile, a self-attention mechanism is introduced to model the spatial dependency across image regions. Extensive experiments demonstrate that our approach can generate photo-realistic video frames with the background, and exhibit the superiorities on accurate synchronization of lip movements and smooth transition of facial movements.

Vehicle model recognition is a typical fine-grained classification task that has a wide range of application prospects in safe cities and constitutes a research hotspot in the field of computer vision. Vehicles in images can appear at various angles, resulting in large differences in appearance. The existence of “multiviews” renders vehicle model recognition challenging. Recent research on vehicle model recognition has not fully explored the pose information of vehicles in different images, resulting in low model performance. In this study, we use vehicle pose information to solve the multiview vehicle model recognition (MV-VMR) problem and design a convolutional neural network (CNN) model with embedded vehicle pose information, known as the embedding pose CNN (EP-CNN). The proposed model includes two subnetworks: the pose estimation subnetwork (PE-SubNet) and vehicle model classification subnetwork (VMC-SubNet). PE-SubNet extracts the vehicle pose information, including the pose features and vehicle viewpoint. In VMC-SubNet, considering the scale variation of vehicles, an improved squeeze-and-excitation (SE) block, named the MultiSE block is implemented. We embed the vehicle viewpoint into the MultiSE block, which reweighs each channel such that the extracted features elicit different responses to different viewpoints. Subsequently, the pose features and classification features are integrated for classification. Experiments are conducted on the benchmark CompCars web-nature and Stanford Cars datasets. The results demonstrate that the proposed EP-CNN method can achieve higher recognition accuracy than most classic CNN models and several state-of-the-art fine-grained vehicle model classification algorithms. Code has been made available at: https://github.com/HFUT-CV/EP-CNN .

Accurate facial expression recognition is challenging because identity biases introduce large intraclass variations and high interclass similarities. Most existing facial expression recognition approaches are devoted to alleviate the effects of identity. However, based on the theories of cognitive science, psychology, and physiology, this article argues that the identity information is important and can promote expression recognition. Motivated by our investigation of the influences of identity on facial expression recognition, this article proposes an identity–expression dual branch network (IE-DBN) for facial expression recognition. First, identity-related features and expression-related features are learnt from the same input facial expression image by two branches respectively. Then, those two features are aggregated with our bilinear module. The bilinear aggregation not only emphasizes the impact of identity but also improves interclass variations and intraclass similarities. The joint training strategy is proposed to regularize the identity-related features. It can force our network to generate expression-guided identity-related features and suppresses negative identity factors in the same time. Experiments on three popular facial expression databases, including two popular posed facial expression databases and one spontaneous facial expression database, show that our IE-DBN outperforms most of the state of the arts, which demonstrates our superiority in facial expression recognition.

In recent years, the rapid development of artificial intelligence, especially deep learning technology, makes machine learning have application scenarios in the fields of power system stability analysis, coordination along with scheduling and load forecasting. This paper designs an emotional deep learning programming controller (EDLPC) for automatic voltage control of power systems. The designed EDLPC contains an emotional deep neural network (EDNN) structure and an artificial emotional Q-learning algorithm. Besides, a specially defined proportional-integral-derivative (PID) controller is added to the deep neural networks (DNNs) structure as the actuator of an EDNN to realize the automatic tuning of PID controller parameters. In terms of control, the controller combines the advantages of the EDNN and PID controller, meanwhile adopts a reinforcement learning algorithm to optimize the parameters. From the perspective of reinforcement learning, embedding prior knowledge into the output instructions of EDNN is helpful to weaken the fitting problem in the training process. Compared with the outputs of the DNN and Q-learning algorithm under the two cases, the EDLPC could gain the highest control performance with smaller voltage deviations. The simulation results verify the feasibility and effectiveness of the proposed method for automatic voltage control of power systems.

We address the issues of 3D head pose estimation and face modeling from a depth image. Given a depth image, random forests are effective for estimating the location and orientation of a person's head. However, the accuracy of the estimation is not high enough. We propose using corrected regression votes. The corrected votes are obtained by considering the cooperation of all trees, leading to significant improvement of head pose estimation accuracy. Based on the head pose estimator, we present a face modeling system. In our system, the face model is generated by aligning a deformable face model to a depth image using iterative closest point (ICP) algorithm. The novelty of our approach is that an optimal weight for each vertex is incorporated into the ICP algorithm with point to plane constraints. Experiments show that our system can automatically estimate the head pose and generate a realistic face model from a single depth image. We also provide a detailed evaluation that shows the benefits of our approach.