This paper presents a new multimodal dataset Moodo that can aid the development of affective music information retrieval systems. Moodo’s main novelties are a multimodal approach that links emotional and color perception to music and the inclusion of user context. Analysis of the dataset reveals notable differences in emotion-color associations and their valence-arousal ratings in non-music and music context. We also show differences in ratings of perceived and induced emotions, especially for those with perceived negative connotation, as well as the influence of genre and user context on perception of emotions. By applying an intermediate data fusion model, we demonstrate the importance of user profiles for predictive modeling in affective music information retrieval scenarios.

Visual information complexity of automotive head-up displays (HUDs) may increase cognitive load and reduce driver performance in critical situations. This study examined whether physiological indicators of cognitive load could predict the type of HUD while driving. Physiological signals of heart rate variability (HRV), electrodermal activity (EDA), skin temperature, and pupil dilation were recorded from 28 participants using a motion-based driving simulator. Two types of HUD with different information complexities were compared: baseline HUD and augmented reality HUD. Heart rate and EDA were processed to generate standardized biomedical features. Two separate sets of features were created for pupil dilation and skin temperature using time-series analysis and basic statistics. The fusion of physiological signals was used to test the impact of different signal combinations on classification performance. Three gradient boosting classifiers LightGBM (LGBM), HistGradientBoostingClassifier (HGBC), and XGBoost (XGB) were trained on physiological signals to predict the type of HUD. Classifiers based on the fusion of HRV, EDA and time-series features for skin temperature and pupil dilation achieved moderate performance, with average AUC ROC scores of XGB=0.67, LGBM=0.69, and HGBC=0.70. Classifiers based on the fusion of HRV, EDA, and the basic statistical features for skin temperature and pupil dilation achieved notable overall improvement in performance, with an average AUC ROC score = 0.76 for all three classifiers and the best score of LGBM=0.96, XGB=0.96, and HGBC=0.98. These results demonstrate the potential of physiological signals for modeling HUD-induced cognitive load and dynamically regulating its effects in real-time.

This study investigated the use of affect and physiological signals of heart rate, electrodermal activity, pupil dilation, and skin temperature to classify advertising engagement. The ground truth for the affective and behavioral aspects of ad engagement was collected from 53 young adults using the User Engagement Scale. Three gradient-boosting classifiers, LightGBM (LGBM), HistGradientBoostingClassifier (HGBC), and XGBoost (XGB), were used along with signal fusion to evaluate the performance of different signal combinations as predictors of engagement. The classifiers trained on the fusion of skin temperature, valence, and tiredness (features n = 5) performed better than those trained on all signals (features n = 30). The average AUC ROC scores for the fusion set were XGB = 0.68 (0.10), LGBM = 0.69 (0.07), and HGBC = 0.70 (0.11), compared to the lower scores for the set of all signals (XGB = 0.65 (0.11), LGBM = 0.66 (0.11), HGBC = 0.64 (0.10)). The results also show that the signal fusion set based on skin temperature outperforms the fusion sets of the other three signals. The main finding of this study is the role of specific physiological signals and how their fusion aids in more effective modeling of ad engagement while reducing the number of features.

The paper presents the development of EthnoMuse: multimedia digital library of Slovenian folk music and dance culture. The main scope of the project concerns the digitization of production and post-production processes that relate to collecting, documenting and archiving of folk heritage and development of multimedia applications for various content types (folk song, music and dance) and formats (image, audio, video, notation, MIDI etc.). The main objective of this paper is to discuss the former, focusing on the conceptual design of a flexible data model the archive is based on. We also briefly describe the tools developed to support the workflow of researchers involved in collecting and archiving of folk music related contents.

The article addresses modelling of consumer engagement in video advertising based on automatically derived low-level video features. The focus is on a young consumer group (18–24 years old) that uses ad-supported online streaming more than any other group. The reference ground truth for consumer engagement was collected in an online crowdsourcing study (N = 150 participants) using the User Engagement Scale-Short Form (UES-SF). Several aspects of consumer engagement were modeled: focused attention, aesthetic appeal, perceived usability, and reward. The contribution of low-level video features was assessed using both the linear and nonlinear models. The best predictions were obtained for the UES-SF dimension Aesthetic Appeal ($R^2=0.35$) using a nonlinear model. Overall, the results show that several video features are statistically significant in predicting consumer engagement with an ad. We have identified linear relations with Lighting Key and quadratic relations with Color Variance and Motion features ($p<0.02$). However, their explained variance is relatively low (up to 25%).