Emotions play a critical role in our daily lives, so the understanding and recognition of emotional responses is crucial for human research. Affective computing research has mostly used non-immersive two-dimensional (2D) images or videos to elicit emotional states. However, immersive virtual reality, which allows researchers to simulate environments in controlled laboratory conditions with high levels of sense of presence and interactivity, is becoming more popular in emotion research. Moreover, its synergy with implicit measurements and machine-learning techniques has the potential to impact transversely in many research areas, opening new opportunities for the scientific community. This paper presents a systematic review of the emotion recognition research undertaken with physiological and behavioural measures using head-mounted displays as elicitation devices. The results highlight the evolution of the field, give a clear perspective using aggregated analysis, reveal the current open issues and provide guidelines for future research.

The purpose of the present study is to investigate whether the effectiveness of a new ad on digital channels (YouTube) can be predicted by using neural networks and neuroscience-based metrics (brain response, heart rate variability and eye tracking). Neurophysiological records from 35 participants were exposed to 8 relevant TV Super Bowl commercials. Correlations between neurophysiological-based metrics, ad recall, ad liking, the ACE metrix score and the number of views on YouTube during a year were investigated. Our findings suggest a significant correlation between neuroscience metrics and self-reported of ad effectiveness and the direct number of views on the YouTube channel. In addition, and using an artificial neural network based on neuroscience metrics, the model classifies (82.9% of average accuracy) and estimate the number of online views (mean error of 0.199). The results highlight the validity of neuromarketing-based techniques for predicting the success of advertising responses. Practitioners can consider the proposed methodology at the design stages of advertising content, thus enhancing advertising effectiveness. The study pioneers the use of neurophysiological methods in predicting advertising success in a digital context. This is the first article that has examined whether these measures could actually be used for predicting views for advertising on YouTube.

Fixation identification is an essential task in the extraction of relevant information from gaze patterns; various algorithms are used in the identification process. However, the thresholds used in the algorithms greatly affect their sensitivity. Moreover, the application of these algorithm to eye-tracking technologies integrated into head-mounted displays, where the subject’s head position is unrestricted, is still an open issue. Therefore, the adaptation of eye-tracking algorithms and their thresholds to immersive virtual reality frameworks needs to be validated. This study presents the development of a dispersion-threshold identification algorithm applied to data obtained from an eye-tracking system integrated into a head-mounted display. Rules-based criteria are proposed to calibrate the thresholds of the algorithm through different features, such as number of fixations and the percentage of points which belong to a fixation. The results show that distance-dispersion thresholds between 1–1.6° and time windows between $0.25\text{–}0.4$ s are the acceptable range parameters, with 1° and $0.25$ s being the optimum. The work presents a calibrated algorithm to be applied in future experiments with eye-tracking integrated into head-mounted displays and guidelines for calibrating fixation identification algorithms

This study compares cognitive and emotional responses to 360-degree vs. static (2D) videos in terms of visual attention, brand recognition, engagement of the prefrontal cortex, and emotions. Hypotheses are proposed based on the interactivity literature, cognitive overload, advertising response model and motivation, opportunity, and ability theoretical frameworks, and tested using neurophysiological tools: electroencephalography, eye-tracking, electrodermal activity, and facial coding. The results revealed that gaze view depends on ad content, visual attention paid being lower in 360-degree FMCG ads than in 2D ads. Brand logo recognition is lower in 360-degree ads than in 2D video ads. Overall, 360-degree ads for durable products increase positive emotions, which carries the risk of non-exposure to some of the ad content. In testing four ads for durable goods and fast-moving consumer goods (FMCG) this research explains the mechanism through which 360-degree video ads outperform standard versions.

Virtual reality (VR) in retailing (V-commerce) has been proven to enhance the consumer experience. Thus, this technology is beneficial to study behavioral patterns by offering the opportunity to infer customers’ personality traits based on their behavior. This study aims to recognize impulsivity using behavioral patterns. For this goal, 60 subjects performed three tasks—one exploration task and two planned tasks—in a virtual market. Four noninvasive signals (eye-tracking, navigation, posture, and interactions), which are available in commercial VR devices, were recorded, and a set of features were extracted and categorized into zonal, general, kinematic, temporal, and spatial types. They were input into a support vector machine classifier to recognize the impulsivity of the subjects based on the I-8 questionnaire, achieving an accuracy of 87%. The results suggest that, while the exploration task can reveal general impulsivity, other subscales such as perseverance and sensation-seeking are more related to planned tasks. The results also show that posture and interaction are the most informative signals. Our findings validate the recognition of customer impulsivity using sensors incorporated into commercial VR devices. Such information can provide a personalized shopping experience in future virtual shops.

There is increasing interest in studies analyzing the influence of technologies that integrate virtual and real-world components on consumer behavior. These technologies include augmented reality, virtual reality and mixed reality. Mixed reality is a user environment in which physical reality and digital content are combined in a way that enables interaction with and among real-world and virtual objects. In spite of previous works related with MR and retails spaces, little is known about how consumers respond to MR features and which elements of the MR-based experience, such as vividness and novelty, impact behavior. In this study, we have explored the relative advantage of mixed reality in retail shopping practices over a traditional-based purchase. Implicit reactions of shoppers when interacting with products with and without MR glasses were compared. The results reveal that participants wearing MR glasses exhibited different patterns of interaction (i.e., frequency and interaction with product duration) that differed from those indicated by participants who did not wear the MR technology. At the level of purchase decision, our results show that the use of MR smart glasses has an impact on decision times that relates to a utilitarian purchase type. Based on participants’ explicit answers to questionnaires, the reported findings further show that the perceived hedonic and utilitarian values of the purchase experience were higher when MR was used, which also affected future purchase intentions and perceived emotional state as reported by consumers’ experience and satisfaction in the context of retail.