Automatic emotion recognition is the process of identifying human emotion from signals such as facial expression, speech, and text. Collecting and labelling such signals is often tedious and many times requires expert knowledge. An effective way to address challenges related to the scarcity of data and lack of human labels, is transfer learning. In this manuscript, we will describe fundamental concepts in the field of transfer learning and review work which has successfully applied transfer learning for automatic emotion recognition. We will finally discuss promising future research directions of transfer learning for improving the generalizability of automatic emotion recognition systems.
30 December 2020 in IEEE Transactions on Affective Computing
Public speaking is a vital skill for making good impressions, effectively exchanging ideas, and influencing others. Yet, public speaking anxiety (PSA) ranks as a top social phobia. Recent advancements in wearable devices and ubiquitous virtual reality (VR) interfaces can help measure and mitigate PSA. This research quantifies PSA through bio-behavioral markers related to individuals' physiological and acoustic characteristics. The effect of virtual reality (VR) training on alleviating PSA is measured through self-reported and bio-behavioral indices. Psychological (e.g., general trait anxiety, personality) and demographic (e.g., age, gender, highest education, native language) traits are examined as moderating factors between bio-behavioral indices and PSA, as well as moderating factors for measuring the VR effectiveness in mitigating PSA. These measures are also used as clustering criteria for stratifying participants in group-based models of PSA. Results indicate the significance of such traits to modeling PSA with the proposed group-based models yielding Spearman's correlation of 0:55 ( $p \lt 0:05$ ) between the actual and predicted outcome. Results further demonstrate that systematic exposure to public speaking in VR can alleviate PSA in terms of both self-reported ( $p \lt 0:05$ ) and physiological ( $p \lt 0:05$ ) indices. Findings from this study will enable researchers to better understand antecedents and causes of PSA and lay the foundation for personalized adaptive feedback for PSA interventions.
3 September 2021 in IEEE Transactions on Affective Computing
Speech-based machine learning (ML) has been heralded as a promising solution for tracking prosodic and spectrotemporal patterns in real-life that are indicative of emotional changes, providing a valuable window into one's cognitive and mental state. Yet, the scarcity of labelled data in ambulatory studies prevents the reliable training of ML models, which usually rely on "data-hungry" distribution-based learning. Leveraging the abundance of labelled speech data from acted emotions, this paper proposes a few-shot learning approach for automatically recognizing emotion in spontaneous speech from a small number of labelled samples. Few-shot learning is implemented via a metric learning approach through a siamese neural network, which models the relative distance between samples rather than relying on learning absolute patterns of the corresponding distributions of each emotion. Results indicate the feasibility of the proposed metric learning in recognizing emotions from spontaneous speech in four datasets, even with a small amount of labelled samples. They further demonstrate superior performance of the proposed metric learning compared to distribution-learning, which is implemented with feedforward neural network fine-tuning. Findings from this work provide a foundation for the ambulatory tracking of human emotion in spontaneous speech contributing to the real-life assessment of mental health degradation outcomes.
Current state-of-the-art residential irrigation systems, such as WaterMyYard, rely on rainfall data from nearby weather stations to adjust irrigation amounts. However, the accuracy of rainfall data is compromised by the limited spatial resolution of rain gauges and the significant variability of hyperlocal rainfall, leading to substantial water waste. To improve irrigation efficiency, we developed a cost-effective irrigation system, dubbed ERIC, which employs machine learning models to estimate rainfall from commodity doorbell camera footage and optimizes irrigation schedules without human intervention. Specifically, we: a) designed novel visual and audio features with lightweight neural network models to infer rainfall from the camera at the edge, preserving user privacy; b) built a complete end-to-end irrigation system on Raspberry Pi 4, costing only $75. We deployed the system across five locations (collecting over 750 hours of video) with varying backgrounds and light conditions. Comprehensive evaluation validates that ERIC achieves state-of-the-art rainfall estimation performance (~ 5mm/day), saving 9,112 gallons/month of water, translating to $28.56/month in utility savings. Data and code are available at https://github.com/LENSS/ERIC-BuildSys2024.git.
25 April 2026 in ACM Transactions on Sensor Networks
Current state-of-the-art residential irrigation systems, such as WaterMyYard, rely on rainfall data from nearby weather stations to adjust irrigation amounts. However, the precision of rainfall data is compromised by the limited spatial resolution of the rain gauges and the significant variability of hyperlocal rainfall, resulting in substantial water waste. To improve irrigation efficiency, we developed a cost-effective irrigation system, dubbed ERIC, which employs machine learning models to estimate rainfall from commodity doorbell camera footage and optimizes irrigation schedules without human intervention. Specifically, we: (a) designed novel visual and audio features with lightweight neural network models to robustly infer rainfall from the camera at the edge, preserving user privacy; (b) built a complete end-to-end irrigation system on Raspberry Pi 4, costing only $75. We deployed the system to five locations (collecting more than 750 hours of video) with diverse environmental backgrounds and varying camera placements. Our realistic evaluation validates that ERIC achieves state-of-the-art rainfall estimation performance (~5 mm/day), saving 9,112 gallons/month of water, translating to $28.56/month in utility savings. Data and code are available at https://github.com/LENSS/ERIC-BuildSys2024.git. 1
This study investigates explainable machine learning algorithms for identifying depression from speech. Grounded in evidence from speech production that depression affects motor control and vowel generation, pre-trained vowel-based embeddings, that integrate semantically meaningful linguistic units, are used. Following that, an ensemble learning approach decomposes the problem into constituent parts characterized by specific depression symptoms and severity levels. Two methods are explored: a "bottom-up" approach with 8 models predicting individual Patient Health Questionnaire-8 (PHQ-8) item scores, and a "top-down" approach using a Mixture of Experts (MoE) with a router module for assessing depression severity. Both methods depict performance comparable to state-of-the-art baselines, demonstrating robustness and reduced susceptibility to dataset mean/median values. System explainability benefits are discussed highlighting their potential to assist clinicians in depression diagnosis and screening.