To improve the human-computer interaction (HCI) to be as good as human-human interaction, building an efficient approach for human emotion recognition is required. These emotions could be fused from several modalities such as facial expression, hand gesture, acoustic data, and biophysiological data. In this paper, we address the frame-based perception of the universal human facial expressions (happiness, surprise, anger, disgust, fear, and sadness), with the help of several geometrical features. Unlike many other geometry-based approaches, the frame-based method does not rely on prior knowledge of a person-specific neutral expression; this knowledge is gained through human intervention and not available in real scenarios. Additionally, we provide a method to investigate the performance of the geometry-based approaches under various facial point localization errors. From an evaluation on two public benchmark datasets, we have found that using eight facial points, we can achieve the state-of-the-art recognition rate. However, this state-of-the-art geometry-based approach exploits features derived from 68 facial points and requires prior knowledge of the person-specific neutral expression. The expression recognition rate using geometrical features is adversely affected by the errors in the facial point localization, especially for the expressions with subtle facial deformations.

Head pose estimation is a crucial initial task for human face analysis, which is employed in several computer vision systems, such as: facial expression recognition, head gesture recognition, yawn detection, etc. In this work, we propose a frame-based approach to estimate the head pose on top of the Viola and Jones (VJ) Haar-like face detector. Several appearance and depth-based feature types are employed for the pose estimation, where comparisons between them in terms of accuracy and speed are presented. It is clearly shown through this work that using the depth data, we improve the accuracy of the head pose estimation. Additionally, we can spot positive detections, faces in profile views detected by the frontal model, that are wrongly cropped due to background disturbances. We introduce a new depth-based feature descriptor that provides competitive estimation results with a lower computation time. Evaluation on a benchmark Kinect database shows that the histogram of oriented gradients and the developed depth-based features are more distinctive for the head pose estimation, where they compare favorably to the current state-of-the-art approaches. Using a concatenation of the aforementioned feature types, we achieved a head pose estimation with average errors not exceeding 5:1; 4:6; 4:2 for pitch, yaw and roll angles, respectively.

The automated analysis of facial expression is a long investigated subject in the computer vision community and has been boosted by applications in the field of human computer interaction (HCI). Besides mapping of facial expressions to basic emotion categories, what is often of limited use for HCI due to sparse occurrence of real emotions, other approaches have been proposed to transform facial expression to the two dimensional so-called valence arousal space. With these affective user state parameters available, the course of the interaction can basically be guided smarter, i.e. the computer can provide help to an apparently confused user. However, it has been shown that the valence arousal space transformation can be impaired due to inaccuracies in image based feature extraction. In this article we present an advanced method using image processing and 3-D computer vision technology that on the one hand suppresses this problem through hierarchical analysis. Further, our concept enables the assignment of an intensity level of the affective state, which can be a valuable parameter for the interaction. In this paper we give details on the system concept with the different processing steps and respective results. By the application of our method we achieve improvement of facial expression recognition compared to other state-of-the-art methods. In particular we can distinguish roughly 15 percent more classes while maintaining the high recognition rate.