Current methods for Music Emotion Recognition (MER) face challenges in effectively extracting features sensitive to emotions, especially those rich in temporal detail. Moreover, the narrow scope of music-related modalities impedes data integration from multiple sources, while including multiple modalities often leads to redundant information, which can degrade performance. To address these issues, we propose a lightweight framework for music emotion recognition that improves the extraction of features that are both sensitive to emotions and rich in temporal information and that integrates data from both audio and MIDI modalities while minimizing redundancy. Our approach involves developing two innovative unimodal encoders to learn embeddings from audio and MIDI-like features. Additionally, we introduce a Bi-modal Fusion Attention Model (BFAM) that integrates features from low-level to high-level semantic information across different modalities. Experimental evaluations on the EMOPIA and VGMIDI datasets show that our unimodal networks achieve accuracies that are 6.1% and 4.4% higher than baseline algorithms for MIDI and audio on the EMOPIA dataset, respectively. Furthermore, our BFAM achieves a 15.2% improvement in accuracy over the baseline, reaching 82.2%, which underscores its effectiveness for bi-modal MER applications

Spectral clustering is a popular clustering method because it gives a natural way to reduce the dimensionality of data using eigenvectors. It is well known that the performance of spectral clustering could be improved via regularization. Nevertheless, it is hard to cope with the different cases by only one constant regularization parameter. To solve such a problem, a novel regularized spectral clustering method is proposed. Specifically, two modules are integrated in the proposed method. First, under matrix perturbation analysis, we prove that the entropy can be used as a rank score function to reveal the informative eigenvector, and the eigenvector corresponding to the minimal entropy will be the regularization to regularize the data matrix instead of a constant regularization parameter. Second, in order to ensure the perturbation on eigenspace is within the effective range, a perturbation boundary on eigenvectors is given. Numerical results showed that our proposal has superior performance than spectral clustering and k-means algorithm.

With the rise in piano teaching in recent years, many people have joined the ranks of piano learners. However, the high cost of traditional manual instruction and the exclusive one-on-one teaching model have made learning the piano an extravagant endeavor. Most existing approaches, based on the audio modality, aim to evaluate piano players’ skills. Unfortunately, these methods overlook the information contained in videos, resulting in a one-sided and simplistic evaluation of the piano player’s skills. More recently, multimodal-based methods have been proposed to assess the skill level of piano players by using both video and audio information. However, existing multimodal approaches use shallow networks to extract video and audio features, which limits their ability to extract complex spatio-temporal and time-frequency characteristics from piano performances. Furthermore, the fingering and pitch-rhythm information of the piano performance is embedded within the spatio-temporal and time-frequency features, respectively. Therefore, we propose a ResNet-based audio-visual fusion model that is able to extract both the visual features of the player’s finger movement track and the auditory features, including pitch and rhythm. The joint features are then obtained through the feature fusion technique by capturing the correlation and complementary information between video and audio, enabling a comprehensive and accurate evaluation of the player’s skill level. Moreover, the proposed model can extract complex temporal and frequency features from piano performances. Firstly, ResNet18-3D is used as the backbone network for our visual branch, allowing us to extract feature information from the video data. Then, we utilize ResNet18-2D as the backbone network for the aural branch to extract feature information from the audio data. The extracted video features are then fused with the audio features, generating multimodal features for the final piano skill evaluation. The experimental results on the PISA dataset show that our proposed audio-visual fusion model, with a validation accuracy of 70.80% and an average training time of 74.02 s, outperforms the baseline model in terms of performance and operational efficiency. Furthermore, we explore the impact of different layers of ResNet on the model’s performance. In general, the model achieves optimal performance when the ratio of video features to audio features is balanced. However, the best performance achieved is 68.70% when the ratio differs significantly.