Wearable technologies and digital phenotyping foster unique opportunities for designing novel intelligent electronic services that can address various well-being issues in patients with mental disorders (i.e., schizophrenia and bipolar disorder), thus having the potential to revolutionize psychiatry and its clinical practice. In this paper, we present e-Prevention, an innovative integrated system for medical support that facilitates effective monitoring and relapse prevention in patients with mental disorders. The technologies offered through e-Prevention include: (i) long-term continuous recording of biometric and behavioral indices through a smartwatch; (ii) video recordings of patients while being interviewed by a clinician, using a tablet; (iii) automatic and systematic storage of these data in a dedicated Cloud server and; (iv) the ability of relapse detection and prediction. This paper focuses on the description of the e-Prevention system and the methodologies developed for the identification of feature representations that correlate with and can predict psychopathology and relapses in patients with mental disorders. Specifically, we tackle the problem of relapse detection and prediction using Machine and Deep Learning techniques on all collected data. The results are promising, indicating that such predictions could be made and leading eventually to the prediction of psychopathology and the prevention of relapses.

Research related to computational modeling for machine-based understanding requires ground truth data for training, content analysis, and evaluation. In this paper, we present a multimodal video database, namely COGNIMUSE, annotated with sensory and semantic saliency, events, cross-media semantics, and emotion. The purpose of this database is manifold; it can be used for training and evaluation of event detection and summarization algorithms, for classification and recognition of audio-visual and cross-media events, as well as for emotion tracking. In order to enable comparisons with other computational models, we propose state-of-the-art algorithms, specifically a unified energy-based audio-visual framework and a method for text saliency computation, for the detection of perceptually salient events from videos. Additionally, a movie summarization system for the automatic production of summaries is presented. Two kinds of evaluation were performed, an objective based on the saliency annotation of the database and an extensive qualitative human evaluation of the automatically produced summaries, where we investigated what composes high-quality movie summaries, where both methods verified the appropriateness of the proposed methods. The annotation of the database and the code for the summarization system can be found at http://cognimuse.cs.ntua.gr/database .

Background: Digital phenotyping refers to the objective measurement of human behavior via devices such as smartphones or watches and constitutes a promising advancement in personalized medicine. Digital phenotypes derived from heart rate, mobility, or sleep schedule data have been used in psychiatry to either diagnose individuals with psychotic disorders or to predict relapse as a binary outcome. Machine learning models so far have achieved predictive accuracies that are significant but not large enough for clinical applications. This could hinge on broad clinical definitions, which encompass heterogeneous symptom and sign ensembles, thus hindering accurate classification. The 5-factor model for the Positive and Negative Syndrome Scale (PANSS), which entails 5 independently varying dimensions, is thought to better capture symptom variability. Using the specific definitions of this refined clinical taxonomy in combination with digital phenotypes could yield more precise results. Objective: This study aims to investigate potential links between digital phenotypes and each dimension of the 5-factor PANSS model. We also assess whether clinical, demographic, and medication variables confound said reactions. Methods: In the e-Prevention study, heart rate, accelerometer, gyroscope, and sleep schedule data were continuously collected via smartwatch for a maximum of 26 months in 38 patients with psychotic spectrum disorders. Obtaining the mean and SD for each patient-month resulted in a database consisting of more than 740 monthly data points. A linear mixed model analysis was used to ascertain connections between monthly aggregated heart rate and mobility features and the 5 symptom dimension scores of PANSS, obtained during monthly clinical interviews. Results: An increase in positive symptoms was associated with a decrease in heart interpulse variation during sleep ( t _570.7 =−3.3 , P <.001, f ² =0.021), while an increase in negative symptoms was associated with a decrease in accelerometer (mean: t _22.1 =−3.1 , P =.005, f ² =0.042; SD: t ₂₀ =−2.4 , P =.03, f ² =0.019), gyroscope (mean: t _22.9 =−2.8 , P =.01, f ² =0.016), and locomotive motor activity ( t _17.2 =−2.4 , P =.03, f ² =0.016) during wakefulness. An increase in accelerometer (mean: t _564.4 =2.8 , P =.005, f ² =0.017; SD: t _551.6 =2.5, P =.01, f ² =0.015) and gyroscope (mean: t _564.5 =3.2 , P =.001, f ² =0.022; SD: t _569.2 =2.8 , P =.005, f ² =0.017) motor activity during sleep was related to an increase in depression/anxiety symptoms as well as excitement/hostility symptoms (accelerometer SD: t _469.7 =3.2 , P =.002, f ² =0.031; gyroscope mean: t ₄₉₇ =2.3 , P =.03, f ² =0.013; SD: t _507.7 =3.2 , P =.001, f ² =0.029). Excitement/hostility symptoms were further associated with an increase in normalized heart rate during sleep ( t _368.2 =3.2, P =.001, f ² =0.044) and reduced sleep:wake ratio ( t ₅₆₂ =−2.7 , P =.007, f ² =0.013). An increase in cognitive/disorganization symptoms was related to a decrease in the SD of normalized heart rate during wakefulness ( t _574.5 =−3.5 , P <.001, f ² =0.013). Conclusions: This study provides evidence that biological changes assessed by continuous measurement of digital phenotypes could be characteristic of specific symptom clusters rather than entire diagnostic categories of psychotic disorders. These results support the use of digital phenotypes not only as a means for remote patient monitoring but also as concrete targets for biomarker research in psychotic disorders.

The original version of this chapter was revised. The first name of one of the authors inadvertantly contained a typo. The author’s first name has been corrected to “Niki.”

This paper presents an overview of the e-Prevention: Person Identification and Relapse Detection Challenge, which was an open call for researchers at ICASSP-2023. The challenge aimed at the analysis and processing of long-term continuous recordings of biosignals recorded from wearable sensors, namely accelerometers, gyroscopes and heart rate monitors embedded in smartwatches, as well as sleep information and daily step counts, in order to extract high-level representations of the wearer's activity and behavior, termed as digital phenotypes. Specifically, with the goal of analyzing the ability of these digital phenotypes in quantifying behavioral patterns, two tasks were evaluated in two distinct tracks: 1) Identification of the wearer of the smartwatch, and 2) Detection of psychotic relapses in patients in the psychotic spectrum. The longterm data that have been used in this challenge have been acquired during the course of the e-Prevention project [1] (https://eprevention.gr/), an innovative integrated system for medical support that facilitates effective monitoring and relapse prevention in patients with mental disorders. Two baseline systems, one for each task, were described and the validation scores for both tasks were provided to the participants. Herein, we present an overview of the approaches and methods as well as the performance analysis and the results of the 5-top ranked participating teams, which in track 1 achieved accuracy results between 91%–95%, while in track 2 mean PR- and ROC-AUC scores between 0.6051 and 0.6489 were obtained. Finally, we also make the datasets publicly available at https://robotics.ntua.gr/eprevention-sp-challenge/ .