In this paper we address the problem of multi-cue affect recognition in challenging scenarios such as child-robot interaction. Towards this goal we propose a method for automatic recognition of affect that leverages body expressions alongside facial ones, as opposed to traditional methods that typically focus only on the latter. Our deep-learning based method uses hierarchical multi-label annotations and multi-stage losses, can be trained both jointly and separately, and offers us computational models for both individual modalities, as well as for the whole body emotion. We evaluate our method on a challenging child-robot interaction database of emotional expressions collected by us, as well as on the GEMEP public database of acted emotions by adults, and show that the proposed method achieves significantly better results than facial-only expression baselines.

Wearable technologies and digital phenotyping foster unique opportunities for designing novel intelligent electronic services that can address various well-being issues in patients with mental disorders (i.e., schizophrenia and bipolar disorder), thus having the potential to revolutionize psychiatry and its clinical practice. In this paper, we present e-Prevention, an innovative integrated system for medical support that facilitates effective monitoring and relapse prevention in patients with mental disorders. The technologies offered through e-Prevention include: (i) long-term continuous recording of biometric and behavioral indices through a smartwatch; (ii) video recordings of patients while being interviewed by a clinician, using a tablet; (iii) automatic and systematic storage of these data in a dedicated Cloud server and; (iv) the ability of relapse detection and prediction. This paper focuses on the description of the e-Prevention system and the methodologies developed for the identification of feature representations that correlate with and can predict psychopathology and relapses in patients with mental disorders. Specifically, we tackle the problem of relapse detection and prediction using Machine and Deep Learning techniques on all collected data. The results are promising, indicating that such predictions could be made and leading eventually to the prediction of psychopathology and the prevention of relapses.

Agricultural robotics is an up and coming field which deals with the development of robotic systems able to tackle a multitude of agricultural tasks efficiently. The case of interest, in this work, is mushroom collection in industrial mushroom farms. Developing such a robot, able to select and out-root a mushroom, requires delicate actions that can only be conducted if a well-performing perception module exists. Specifically, one should accurately detect the 3D pose of a mushroom in order to facilitate the smooth operation of the robotic system. In this work, we develop a vision module for 3D pose estimation of mushrooms from multi-view point clouds using multiple RealSense active–stereo cameras. The main challenge is the lack of annotation data, since 3D annotation is practically infeasible on a large scale. To address this, we developed a novel pipeline for mushroom instance segmentation and template matching, where a 3D model of a mushroom is the only data available. We evaluated, quantitatively, our approach over a synthetic dataset of mushroom scenes, and we, further, validated, qualitatively, the effectiveness of our method over a set of real data, collected by different vision settings.

Background: Digital phenotyping refers to the objective measurement of human behavior via devices such as smartphones or watches and constitutes a promising advancement in personalized medicine. Digital phenotypes derived from heart rate, mobility, or sleep schedule data have been used in psychiatry to either diagnose individuals with psychotic disorders or to predict relapse as a binary outcome. Machine learning models so far have achieved predictive accuracies that are significant but not large enough for clinical applications. This could hinge on broad clinical definitions, which encompass heterogeneous symptom and sign ensembles, thus hindering accurate classification. The 5-factor model for the Positive and Negative Syndrome Scale (PANSS), which entails 5 independently varying dimensions, is thought to better capture symptom variability. Using the specific definitions of this refined clinical taxonomy in combination with digital phenotypes could yield more precise results. Objective: This study aims to investigate potential links between digital phenotypes and each dimension of the 5-factor PANSS model. We also assess whether clinical, demographic, and medication variables confound said reactions. Methods: In the e-Prevention study, heart rate, accelerometer, gyroscope, and sleep schedule data were continuously collected via smartwatch for a maximum of 26 months in 38 patients with psychotic spectrum disorders. Obtaining the mean and SD for each patient-month resulted in a database consisting of more than 740 monthly data points. A linear mixed model analysis was used to ascertain connections between monthly aggregated heart rate and mobility features and the 5 symptom dimension scores of PANSS, obtained during monthly clinical interviews. Results: An increase in positive symptoms was associated with a decrease in heart interpulse variation during sleep ( t _570.7 =−3.3 , P <.001, f ² =0.021), while an increase in negative symptoms was associated with a decrease in accelerometer (mean: t _22.1 =−3.1 , P =.005, f ² =0.042; SD: t ₂₀ =−2.4 , P =.03, f ² =0.019), gyroscope (mean: t _22.9 =−2.8 , P =.01, f ² =0.016), and locomotive motor activity ( t _17.2 =−2.4 , P =.03, f ² =0.016) during wakefulness. An increase in accelerometer (mean: t _564.4 =2.8 , P =.005, f ² =0.017; SD: t _551.6 =2.5, P =.01, f ² =0.015) and gyroscope (mean: t _564.5 =3.2 , P =.001, f ² =0.022; SD: t _569.2 =2.8 , P =.005, f ² =0.017) motor activity during sleep was related to an increase in depression/anxiety symptoms as well as excitement/hostility symptoms (accelerometer SD: t _469.7 =3.2 , P =.002, f ² =0.031; gyroscope mean: t ₄₉₇ =2.3 , P =.03, f ² =0.013; SD: t _507.7 =3.2 , P =.001, f ² =0.029). Excitement/hostility symptoms were further associated with an increase in normalized heart rate during sleep ( t _368.2 =3.2, P =.001, f ² =0.044) and reduced sleep:wake ratio ( t ₅₆₂ =−2.7 , P =.007, f ² =0.013). An increase in cognitive/disorganization symptoms was related to a decrease in the SD of normalized heart rate during wakefulness ( t _574.5 =−3.5 , P <.001, f ² =0.013). Conclusions: This study provides evidence that biological changes assessed by continuous measurement of digital phenotypes could be characteristic of specific symptom clusters rather than entire diagnostic categories of psychotic disorders. These results support the use of digital phenotypes not only as a means for remote patient monitoring but also as concrete targets for biomarker research in psychotic disorders.

This paper proposes a novel lightweight visual perception system with Incremental Learning (IL), tailored to child–robot interaction scenarios. Specifically, this encompasses both an action and emotion recognition module, with the former wrapped around an IL system, allowing novel actions to be easily added. This IL system enables the tutor aspiring to use robotic agents in interaction scenarios to further customize the system according to children’s needs. We perform extensive evaluations of the developed modules, achieving state-of-the-art results on both the children’s action BabyRobot dataset and the children’s emotion EmoReact dataset. Finally, we demonstrate the robustness and effectiveness of the IL system for action recognition by conducting a thorough experimental analysis for various conditions and parameters.