Captioning is the process of transcribing speech and acoustical information into text to help deaf and hard of hearing people accessing to the auditory track of audiovisual media. In addition to the verbal transcription, it includes information such as sound effects, speaker identification, or music tagging. However, it just takes into account a limited spectrum of the whole acoustic information available in the soundtrack, and hence, an important amount of emotional information is lost when attending just to the normative compliant captions. In this article, it is shown, by means of behavioral and EEG measurements, how emotional information related to sounds and music used by the creator in the audiovisual work is perceived differently by normal hearing group and hearing disabled group when applying standard captioning. Audio and captions activate similar processing areas, respectively, in each group, although not with the same intensity. Moreover, captions require higher activation of voluntary attentional circuits, as well as language-related areas. Captions transcribing musical information increase attentional activity, instead of emotional processing.
Standard captioning for the deaf and hard of hearing people cannot transmit the emotional information that music provides in support of the narrative in audio-visual media. We explore an alternative method using vibrotactile stimulation as a possible channel to transmit the emotional information contained in an audio-visual soundtrack and, thus, elicit a greater emotional reaction in hearing-impaired people. To achieve this objective, we applied two one-minute videos that were based on image sequences that were unassociated with dramatic action, maximizing the effect of the music and vibrotactile stimuli. While viewing the video, using EEG we recorded the brain activity of 9 female participants with normal hearing, and 7 female participants with very severe and profound hearing loss. The results show that the same brain areas are activated in participants with normal hearing watching the video with the soundtrack, and in participants with hearing loss watching the same video with a soft and rhythmic vibrotactile stimulation on the palm and fingertips, although in different hemispheres. These brain areas (auditory cortex, superior temporal cortex, medial frontal cortex, inferior frontal gyrus, superior temporal pole and insula) have been consistently reported as areas involved in the emotional perception of music. We conclude that vibrotactile stimuli can generate cortex activation while watching audio-visual media in a similar way to sound. Thus, a further in-depth study of the possibilities of these stimuli can contribute to an alternative subtitling channel for enriching the audiovisual experience of hearing-impaired people.
The ability of music to induce emotions has been arousing a lot of interest in recent years, especially due to the boom in music streaming platforms and the use of automatic music recommenders. Music Emotion Recognition approaches are based on combining multiple audio features extracted from digital audio samples and different machine learning techniques. In these approaches, neuroscience results on musical emotion perception are not considered. The main goal of this research is to facilitate the automatic subtitling of music. The authors approached the problem of automatic musical emotion detection in movie soundtracks considering these characteristics and using scientific musical databases, which have become a reference in neuroscience research. In the experiments, the Constant-Q-Transform spectrograms, the ones that best represent the relationships between musical tones from the point of view of human perception, are combined with Convolutional Neural Networks. Results show an efficient emotion classification model for 2-second musical audio fragments representative of intense basic feelings of happiness, sadness, and fear. Those emotions are the most interesting to be identified in the case of movie music captioning. The quality metrics have demonstrated that the results of the different models differ significantly and show no homogeneity. Finally, these results pave the way for an accessible and automatic captioning of music, which could automatically identify the emotional intent of the different segments of the movie soundtrack.
La característica más representativa de la música es su capacidad de generar emoción. Pero ¿por qué la música emociona? En este artículo mostramos los conocimientos actuales de la teoría musical y la neurociencia que intentan explicar las relaciones que existen entre la música y las emociones. En primer lugar, se repasan los conocimientos actuales sobre el procesamiento de los sonidos musicales a nivel cerebral y las posibles explicaciones del origen de la emoción musical, así como la contribución de los distintos parámetros musicales a la generación de emociones, considerando estructuras musicales de nuestro mundo occidental. En segundo lugar, se presenta el canal táctil como un posible canal de transmisión de la emoción musical análogo al canal auditivo, pero con más limitaciones en la discriminación de frecuencias. Este acercamiento se produce desde la profunda fascinación que ejerce la música, con la esperanza de encontrar vías para explicar la transmisión de la emoción musical desde un canal alternativo.
In recent years, with the development of haptic technologies, the investigation of the potential of vibrotactile perception of musical parameters has attracted much interest. The possibility of vibrotactile musical note discrimination has already been studied. In this study, we approach the problem of vibrotactile perception of musical consonant and dissonant tone relationships, essential components of Western tonal music. Thirty participants were asked to distinguish between consonant and dissonant intervals presented in two different conditions: the Auditory Condition and the Vibrotactile Condition (through tactile stimulation). The stimuli were occidental tonal piano music intervals considered from the point of view of a musical theory perfect consonant or dissonant interval. The results show that consonant and dissonant musical intervals can be perceived at the tactile level and that there is no significant difference in the number of intervals correctly recognised in the Vibrotactile Condition and the Auditory Condition in participants who have no musical training. The consonance/dissonance perception shows some differences in both conditions, with vibrotactile perception being more accurate with larger intervals of more than ten semitones. In the Auditory Condition, it is related to the number of semitones, becoming more sensitive from eleven semitones onwards, and the type of interval, possibly due to the influence of auditory musical training. These results open up the possibility of transmitting other tonal musical characteristics; through tactile stimulation the possibility of transmitting the melodic and harmonic basis of Western music vibrotactically opens up, offering a wide range of options for investigation.