Video-based dynamic facial emotion recognition (FER) is a challenging task, as one must capture and distinguish tiny facial movements representing emotional changes while ignoring the facial differences of different objects. Recent state-of-the-art studies have usually adopted more complex methods to solve this task, such as large-scale deep learning models or multimodal analysis with reference to multiple sub-models. According to the characteristics of the FER task and the shortcomings of existing methods, in this paper we propose a lightweight method and design three attention modules that can be flexibly inserted into the backbone network. The key information for the three dimensions of space, channel, and time is extracted by means of convolution layer, pooling layer, multi-layer perception (MLP), and other approaches, and attention weights are generated. By sharing parameters at the same level, the three modules do not add too many network parameters while enhancing the focus on specific areas of the face, effective feature information of static images, and key frames. The experimental results on CK+ and eNTERFACE’05 datasets show that this method can achieve higher accuracy.

Three-dimensional face analysis is now a hot research topic in computer vision. The acquisition of 3D faces requires acquiring multi-view face and obtaining them by point cloud registration. In this paper, a deep learning method is introduced in the multi-view RGBD depth sensor system to acquire matching point pairs of image data. Thus, 3D spatial point pairs are obtained. Then, the initial poses between the point clouds of multi-view faces are obtained by rigid body transformation estimation, and the coarse registration of the point clouds is completed. On this basis, for fine registration of multi-view 3D face point clouds using the Iterative Closest Point (ICP) algorithm, this paper proposes a method to convert partially overlapping point cloud registration into sub-region point cloud registration through facial region positioning. In this letter, the maximum value of RMSE for the experimental results of point cloud coarse registration is 2.26 mm with the mean and standard deviation of 1.18 ± 0.33 mm; the maximum value of RMSE for the experimental results of point cloud fine registration is 1.47 mm with the mean and standard deviation of 0.91 ± 0.14 mm. It is confirmed that the proposed method is very robust and effective for the registration of multi-view 3D face point clouds.

Head pose estimation (HPE) is a key step in computation and quantification of 3D facial features and has a significant impact on the precision and accuracy of measurements. High-precision HPE is the basis for standardized facial data collection and analysis. The Camper’s plane is the standard (baseline) plane commonly used by anthropologists for head and face research, but there is no research on automatic positioning of the Camper’s plane using color and depth cameras. This paper presents a high-accuracy method for Camper’s plane localization and HPE based on multi-view RGBD depth sensors. The 3D facial point clouds acquired by the multi-view RGBD depth sensors are aligned to obtain a complete 3D face. Keypoint RCNN is used for facial keypoint detection to obtain facial landmarks. A method is proposed to build a general face datum model based on a self-built dataset. The head pose is estimated by applying rigid body transformation to an individual 3D face and the general 3D face model. In order to verify the accuracy of Camper’s plane localization and HPE, 102 cases of 3D facial data and experiments were collected and conducted. The tragus and nasal alar points are localized to within 7 pixels (about 0.83 cm) and the average accuracy of the three dimensions of Camper’s plane identified is 0.87°, 0.64° and 0.47° respectively. The average accuracies of the three dimensions of HPE were 1.17°, 0.90° and 0.97. The experiment results demonstrate the effectiveness of the method for Camper’s plane localization and HPE.

Fusion of infrared (IR) and visible sensor images is the widely applied. The registration is the basis for sensor fusion and fusion methods are very sensitive to the level of registration accuracy. However, the different imaging systems of the both lead to quite different image characteristics in a same scene and significant misalignment due to differences in field of view, lens distortion and other camera characteristics. So that, the registering accurately the both of sensor images is very challenging. A method to improve the accuracy of image registration is proposed in this paper. The method is based a self-designed long-wave IR/visible dual-band imaging system for capturing simultaneously both of images, which is not only with synchronous focusing and optical registration as much as possible, but also with a sliding potentiometer that record the voltage corresponding to the focusing distance. At first, an affine transformation of the registration at several different distances is acquired with two calibration board images captured self-designed calibration board by the dual-band imaging system. Then, the affine transformation matrices corresponding to the several finite distance are interpolated to obtain more affine transformation matrices within a certain distance, and an accurate look-up table is established. Last, when the dual-band imaging system is working, the current focal length is read out according to the sliding potentiometer in the system, and then the corresponding affine transformation matrix is searched for image registration. The proposed method is evaluated by comparing the deviation of the corresponding feature point coordinates on both of calibration board images before and after registration. Experimental results show that the proposed method can improve the registration accuracy of IR and visible images at different distances.