Automatic speech emotion recognition (SER) has gained popularity over the last decade and numerous Challenges have emerged. While the latest Challenges have shown that deep neural networks achieve the best results, existing input features are still a bottleneck and cause severe performance degradation in realistic “in-the-wild” scenarios. In this paper, we propose two innovations to tackle this issue. First, we propose to combine the bag-of-audio-words methodology with modulation spectrum features for environmental robustness. Second, we take advantage of the inherent quality-awareness properties of modulation spectrum and propose the use of a quality feature as an additional feature to be used by the speech emotion recognizer. Experiments are conducted with three multi-lingual speech datasets used in recent SER Challenges degraded by different noise sources and levels, and room reverberation. Experimental results show the proposed features i) consistently outperforming benchmark systems, ii) providing complementary information to classical features, hence improving performance with feature fusion, and iii) showing robustness against environment and language mismatch. Moreover, we show that when the proposed system is provided with quality information, further improvements are obtained. Overall, the proposed bag of modulation spectrum features are shown to be a promising candidate for “in-the-wild” SER.
To date, several methods have been explored for the challenging task of cross-language speech emotion recognition, including the bag-of-words (BoW) methodology for feature processing, domain adaptation for feature distribution “normalization”, and data augmentation to make machine learning algorithms more robust across testing conditions. Their combined use, however, has yet to be explored. In this paper, we aim to fill this gap and compare the benefits achieved by combining different domain adaptation strategies with the BoW method, as well as with data augmentation. Moreover, while domain adaptation strategies, such as the correlation alignment (CORAL) method, require knowledge of the test data language, we propose a variant that we term N-CORAL, in which test languages (in our case, Chinese) are mapped to a common distribution in an unsupervised manner. Experiments with German, French, and Hungarian language datasets were performed, and the proposed N-CORAL method, combined with BoW and data augmentation, was shown to achieve the best arousal and valence prediction accuracy, highlighting the usefulness of the proposed method for “in the wild” speech emotion recognition. In fact, N-CORAL combined with BoW was shown to provide robustness across languages, whereas data augmentation provided additional robustness against cross-corpus nuance factors.
Wearable devices are burgeoning, and applications across numerous verticals are emerging, including human performance monitoring, at-home patient monitoring, and health tracking, to name a few. Off-the-shelf wearables have been developed with focus on portability, usability, and low-cost. As such, when deployed in highly ecological settings, wearable data can be corrupted by artifacts and by missing data, thus severely hampering performance. In this technical note, we overview a signal processing representation called the modulation spectrum. The representation quantifies the rate-of-change of different spectral magnitude components and is shown to separate signal from noise, thus allowing for improved quality measurement, quality enhancement, and noise-robust feature extraction, as well as for disease characterization. We provide an overview of numerous applications developed by the authors over the last decade spanning different wearable modalities and list the results obtained from experimental results alongside comparisons with various state-of-the-art benchmark methods. Open-source software is showcased with the hope that new applications can be developed. We conclude with a discussion on possible future research directions, such as context awareness, signal compression, and improved input representations for deep learning algorithms.
Automated building damage assessment from satellite imagery has become increasingly critical for rapid disaster response and humanitarian relief operations. However, current state-of-the-art deep learning models exhibit significant generalization challenges when deployed to geographically and environmentally diverse regions. This study investigates the nature and extent of geographic bias in building damage detection systems, revealing that model performance degradation occurs primarily from geographic and structural characteristics rather than insufficient training data representation. Through a systematic evaluation of top-performing xView2 competition solutions across 17 disaster locations across multiple climate zones, we found that even state-of-the-art models struggle with generalization under geographic shift, particularly for the minor and major damage classes, and exhibit strong geographic biases toward certain regions. In this work, we explore a six-channel Fusion Augmentation strategy that enriches RGB imagery with auxiliary structural enhancement channels, together with supervised fine-tuning and unsupervised CORAL-based domain adaptation for three unseen regions. The experimental results demonstrate a substantial improvement of 7.1% overall F1 score, with notable gains for intermediate damage categories such as minor and major damage. Domain adaptation experiments on three unseen locations show that combining Fusion Augmentation with supervised fine-tuning yields 40.8% and 60.0% improvements over the minor and major classes, while unsupervised CORAL achieves 24.2% and 39.5% improvements over the minor and major damage classes compared to benchmarks. These findings highlight persistent geographic bias and demonstrate that structural feature enhancement combined with domain adaptation is essential for globally deployable damage assessment systems.
Automated sleep staging remains challenging due to the transitional nature of certain sleep stages, particularly N1. In this paper, we explore modulation spectrograms for automatic sleep staging to capture the transitional nature of sleep stages and compare them with conventional benchmark features, such as the Short-Time Fourier Transform (STFT) and the Continuous Wavelet Transform (CWT). We utilized a single-channel EEG (C4–M1) from the DREAMT dataset with subject-independent validation. We stratify participants by the Apnea–Hypopnea Index (AHI) into Normal, Mild, Moderate, and Severe groups to assess clinical generalizability. Our modulation-based framework significantly outperforms STFT and CWT in the Mild and Severe cohorts, while maintaining comparable high performance in the Normal and Moderate AHI groups. Notably, the proposed framework maintained robust performance in severe apnea cohorts, effectively mitigating the degradation observed in standard time–frequency baselines. These findings demonstrate the effectiveness of modulation spectrograms for sleep staging while emphasizing the importance of medical stratification for reliable outcomes in clinical populations.
17 January 2025 in Journal of the Brazilian Computer Society
Agriculture plays a crucial role in Brazil's economy. As the country intensifies its activities in the sector, the use of pesticides also increases. Hence, the risks associated with pesticide-laden food consumption have become a concern for chemistry researchers. An issue affecting regulatory standardization of pesticides in Brazil is the difficulty in translating pesticide names, particularly from English. For example, the word malathion can be translated from English to Portuguese as malatiom or malatião, resulting in inconsistent labeling. This issue extends to the broader problem of translating highly technical terms between languages, in particular for low-resource languages. In this work, we investigate terminological variation in the chemistry of organophosphorus pesticides. Our goal is to study strategies for domain-specific multilingual keyword extraction. To that end, two corpora were built based on pesticide-related scientific documents in Brazilian Portuguese and English, which led to a total of 84 and 210 texts, respectively, representing the low- and high-resource languages in this study. We then assessed 6 methods for keyword extraction: Simple Maths, TF-IDF, YAKE, TextRank, MultipartiteRank, and KeyBERT. We relied on a multilingual contextual BERT embedding to retrieve corresponding pesticide names in the target language. Fine-tuning was also explored to improve the multilingual representation further. Moreover, we evaluated the use of large language models (LLMs) combined with the recent retrieval-augmented generation (RAG) framework. As a result, we found that the contextual approach, combined with fine-tuning, provided the best results, contributing to enhancing Pesticide Terminology Extraction in a multilingual scenario.