This study investigates whether taking genre into account is beneficial for automatic music mood annotation in terms of core affects valence, arousal, and tension, as well as several other mood scales. Novel techniques employing genre-adaptive semantic computing and audio-based modelling are proposed. A technique called the ACTwg employs genre-adaptive semantic computing of mood-related social tags, whereas ACTwg-SLPwg combines semantic computing and audio-based modelling, both in a genre-adaptive manner. The proposed techniques are experimentally evaluated at predicting listener ratings related to a set of 600 popular music tracks spanning multiple genres. The results show that ACTwg outperforms a semantic computing technique that does not exploit genre information, and ACTwg-SLPwg outperforms conventional techniques and other genre-adaptive alternatives. In particular, improvements in the prediction rates are obtained for the valence dimension which is typically the most challenging core affect dimension for audio-based annotation. The specificity of genre categories is not crucial for the performance of ACTwg-SLPwg. The study also presents analytical insights into inferring a concise tag-based genre representation for genre-adaptive music mood analysis.
We present a method for tempo estimation from audio recordings based on signal processing and peak tracking, and not depending on training on ground-truth data. First, an accentuation curve, emphasizing the temporal location and accentuation of notes, is based on a detection of bursts of energy localized in time and frequency. This enables the detection of notes in dense polyphonic texture, while ignoring spectral fluctuation produced by vibrato and tremolo. Periodicities in the accentuation curve are detected using an improved version of autocorrelation function. Hierarchical metrical structures, composed of a large set of periodicities in pairwise harmonic relationships, are tracked over time. In this way, the metrical structure can be tracked even if the rhythmical emphasis switches from one metrical level to another. This approach, compared to all the other participants to the Music Information Retrieval Evaluation eXchange (MIREX) Audio Tempo Extraction competition from 2006 to 2018, is the third best one among those that can track tempo variations. While the two best methods are based on machine learning, our method suggests a way to track tempo founded on signal processing and heuristics-based peak tracking. Moreover, the approach offers for the first time a detailed representation of the dynamic evolution of the metrical structure. The method is integrated into MIRtoolbox, a Matlab toolbox freely available.
Resilience to changes in performance environments is a hallmark of expertise in music performance. Research has shown that skilled non-expert ensemble musicians maintain synchronization when visual interaction between them is disrupted, but that the quality of their expressive body motion changes. Our study extended these findings by testing how an expert string quartet responds to playing conditions that disrupt visual contact. We tested for potential effects on the musicians’ expressive head motion, sound quality, and mental effort. The Danish String Quartet (DSQ), a world-class classical ensemble, performed an excerpt from a Haydn piece five times without an audience, then once for an audience of about 20 people. During the performances without audience, their seating configuration was manipulated to disrupt their audiovisual interaction. Audio, head motion, eye-tracking, and pupillometry data were collected. The DSQ's data were compared to data from a student quartet who completed the same experiment. Our results show that the DSQ maintained the quality of their musical sound and interactive body motion across disruptive and non-disruptive conditions, but mental effort (indexed by pupil size) was greater in non-disruptive conditions. In contrast, the student quartet moved less overall than the DSQ, moved less when they could not see each other, and did not show differences in pupil size across conditions. The quartets spent a similar percentage of performance time watching their co-performers. These findings suggest that the quality of audio and visual components of the DSQ's performance do not require visual interaction to maintain; however, these musicians do interact visually when given the opportunity. This visual interaction stimulates greater mental effort, perhaps reflecting increased social engagement.
The term ``multimodal music dataset'' is often used to describe music-related data that represent music as a multimedia art form and multimodal experience. However, the term ``multimodality'' may mean different things in related disciplines, such as musicology, music psychology, and music technology. This paper proposes a definition of multimodality that works across different music disciplines. Many challenges are related to constructing, evaluating, and using multimodal music datasets. We provide a task-based categorization of these datasets and explore theoretical methodologies aimed at enhancing their future construction. Diverse data preprocessing methods are illuminated, highlighting their contributions to transparent music analysis. Additionally, evaluation metrics, methods, and benchmarks tailored for multimodal music processing tasks are scrutinized, empowering researchers to make informed decisions and facilitating cross-study comparisons.
13 December 2023 in Transactions of the International Society for Music Information Retrieval
The Hardanger fiddle is a variety of the violin used in the folk music of the western and central part of southern Norway. This paper presents a dataset of several hours of recordings of Hardanger fiddle music, with note annotations of onsets, offsets and pitches, provided by the performers themselves. A subset has also been annotated with beat onset positions by the performer as well as three expert musicians. The complexity of the music genre—polyphonic, highly ornamented and with a very irregular pulsation, among other aspects—motivated the design of a new annotation software adapted to these particular needs. Beat annotation in MIR is typically recorded as positions in seconds, without explicit connection with actual musical events. In the context of music where the rhythm is carried by the melodic instrument alone, a more reliable definition of beat onsets consists in associating them with the onsets of the notes that represent the start of each beat. This latter definition of beat onsets reflects that beats are generated from within the flow of played melodic-rhythmic events, which implies that the spacing of beats may be shifting and irregular. This motivated the design of a new method for beat annotation in Hardanger fiddle music based on a selection of notes in the note annotation. Comparisons between annotators through alignment—integrated in the interface—enable them to eventually correct their annotations or observe alternative valid interpretations of any given excerpt. After dedicating a part of the note annotation dataset to the training of a machine learning model, for the task of assessing both note pitch and onset time, an F1 score of 87% can be reached. The beat annotation dataset demonstrates the necessity of developing new beat trackers adapted to Hardanger fiddle music. The dataset as well as the annotation software is made publicly available.