Artificial intelligence (AI) models can sense subjective affective states from facial images. Although recent psychological studies have indicated that dimensional affective states of valence and arousal are systematically associated with facial expressions, no AI models have been developed to estimate these affective states from facial images based on empirical data. We developed a recurrent neural network-based AI model to estimate subjective valence and arousal states from facial images. We trained our model using a database containing participant valence/arousal states and facial images. Leave-one-out cross-validation supported the validity of the model for predicting subjective valence and arousal states. We further validated the effectiveness of the model by analyzing a dataset containing participant valence/arousal ratings and facial videos. The model predicted second-by-second valence and arousal states, with prediction performance comparable to that of FaceReader, a commercial AI model that estimates dimensional affective states based on a different approach. We constructed a graphical user interface to show real-time affective valence and arousal states by analyzing facial video data. Our model is the first distributable AI model for sensing affective valence and arousal from facial images/videos to be developed based on an empirical database; we anticipate that it will have many practical uses, such as in mental health monitoring and marketing research.
1 January 2016 in International Journal of Affective Engineering
Facial expression recognition (FER) has been widely studied since it can be used for various applications. However, most of FER techniques focus on discriminating typical facial expressions such as six basic facial expressions. Spontaneous facial expressions are not limited to such typical ones because the intensity of a facial expression varies depending on the intensity of an emotion. In order to utilize FER for real-world applications, therefore, it is necessary to discriminate slight difference of facial expressions. In this paper, we propose an effective FER method to recognize spontaneous facial expressions using ensemble learning which combines a number of naive Bayes classifiers. In addition, a method to estimate the intensity of facial expression is also proposed by using the classification results of the classifiers. The effectiveness of these methods are evaluated through an FER experiment and an experiment to estimate the intensity of facial expressions using a data set including spontaneous facial expressions.