17 September 2025 in IEEE Access
Contrastive language-audio pre-training (CLAP), which learns audio-language representations by aligning audio and text in a common feature space, has become popular for solving audio tasks. However, CLAP’s audio features lack generalizability, whereas self-supervised learning (SSL) models offer general-purpose features that perform well across diverse audio tasks. We aim to develop a broadly applicable audio representation and hypothesize that a model that learns both general audio and CLAP features should achieve our goal, which we call a general-purpose audio-language representation. To implement our hypothesis, we propose M2D-CLAP, the first approach to jointly learn effective general audio and CLAP features. It extends an SSL masked modeling duo (M2D) by incorporating CLAP and utilizes LLM-based sentence embeddings. The training process consists of multiple stages. In the first stage, generalizable audio features are pre-trained via a multitask objective combining M2D and CLAP, with CLAP leveraging LLM-based semantic embeddings to distill semantic knowledge into them. In the following stages, CLAP features are pre-trained and refined with guidance from the learned audio features. Experiments demonstrated that M2D-CLAP learns high-performing general audio features (e.g., AudioSet mAP of 49.0, SOTA results in music tasks) and CLAP features, thereby enabling a general-purpose audio-language representation.
10 February 2026 in ACM Transactions on Multimedia Computing, Communications, and Applications
This article addresses a new task: distributed multimedia sensor event analysis (DiMSEA). DiMSEA aims to analyze a series of human and machine activities (called “events” in this article) in complex and extensive real-world environments. Since an observation from a single sensor is often missing or fragmented in such an environment, observations from multiple locations and modalities should be integrated to analyze events comprehensively. However, a learning method has yet to be established to extract joint representations that effectively combine such distributed observations. Therefore, we propose guided masked self-distillation modeling (Guided-MELD) for inter-sensor relationship modeling. The basic idea of Guided-MELD is to learn to supplement the information from the masked sensor with information from other sensors needed to detect the event. Guided-MELD is expected to effectively distill fragmented target event information from sensors without over-relying on any specific sensors. To validate the effectiveness of the proposed method in DiMSEA, we recorded two new datasets: MM-Store and MM-Office. These datasets consist of human activities in a convenience store and an office, recorded using distributed cameras and microphones. Experimental results show that the proposed Guided-MELD improves event tagging and detection performance and outperforms conventional inter-sensor relationship modeling methods. Furthermore, the proposed method performed robustly even when sensors were reduced.
12 September 2025 in IEEE Access
Sound Event Localization and Detection (SELD) typically relies on strongly labeled data that specify when, where, and which sounds occur, resulting in high annotation costs. To address this challenge, we present a framework that enables SELD training without spatial labels, termed spatial annotation-free SELD. The core idea is to reinterpret source localization—traditionally handled as a regression problem—as a classification task called spatial instance classification (SIC). SIC first partitions the observation region into sub-regions. For each sub-region, the acoustic signal capturing its sound events is extracted as a spatial instance. Classifying these instances reveals the sub-regions that contain sound events, thereby providing SELD functionality. By solving SIC under weak supervision, we realize spatial annotation-free SELD. SIC does not depend on a specific microphone configuration, and it is expected to be generally applicable as long as the sound in each divided region can be extracted as an instance. We validate the generality of SIC through two contrasting experiments: (i) application to SELD using first-order ambisonics (FOA), and (ii) application to a newly introduced distributed SELD task, where events are localized on a 2D plane using widely spaced microphones. In both settings, SIC effectively trains SELD systems capable of localizing sound events without spatial annotations. The proposed approach, therefore, facilitates the development of practical, spatial-annotation-free SELD across diverse microphone setups.