Background: This paper presents TheraSense, a system developed within the Supporting Mental Health in Young People: Integrated Methodology for cLinical dEcisions and evidence (Smile) and Sensor Enabled Affective Computing for Enhancing Medical Care (SenseCare) projects. TheraSense is designed to enhance teleconsultation services by leveraging deep learning for real-time emotion recognition through facial expressions. It integrates with the Knowledge Management-Ecosystem Portal (SenseCare KM-EP) platform to provide mental health practitioners with valuable emotional insights during remote consultations. Method: We describe the conceptual design of TheraSense, including its use case contexts, architectural structure, and user interface layout. The system’s interoperability is discussed in detail, highlighting its seamless integration within the teleconsultation workflow. The evaluation methods include both quantitative assessments of the video-based emotion recognition system’s performance and qualitative feedback through heuristic evaluation and survey analysis. Results: The performance evaluation shows that TheraSense effectively recognizes emotions in video streams, with positive user feedback on its usability and integration. The system’s real-time emotion detection capabilities provide valuable support for mental health practitioners during remote sessions. Conclusions: TheraSense demonstrates its potential as an innovative tool for enhancing teleconsultation services. By providing real-time emotional insights, it supports better-informed decision-making in mental health care, making it an effective addition to remote telehealth platforms.

Early, intensive, and repetitive physical rehabilitation is critical, but it can be difficult to keep patients motivated and engaged. The use of games in immersive reality makes physiotherapy more enjoyable and engaging. During play, many unpleasant emotions can occur as a result of interacting with the joyful activities of the game. This difficulty can be overcome by adjusting game features to the players' emotions and body gestures. This work contributes to the Sensor Enabled Affective Computing for Enhancing Medical Care (SenseCare) project for remote home healthcare applications, and its related SenseCare KM-EP (Knowledge Management-Ecosystem Portal) Affective Computing (AC) platform. In this paper, we propose web-based Augmented Reality (AR) and Virtual Reality (VR) games for home rehabilitation deploying audiovisual sensors (e.g., Camera, Microphone, Kinect), to monitor the patient's emotional well-being and calculate body estimation gestures during the gameplay session, with the aim of giving a quick return to the therapist. In parallel with SenseCare's methods for audiovisual monitoring of gamers, methods for using VR and AR web-based games using the WebXR API are also introduced, and customer satisfaction was determined through an online survey, showing an 81.8% satisfaction rate.

The T-Rehab System delivers a semi-immersive tele-rehabilitation experience by integrating Affective Computing (AC) through facial expression analysis and contactless heartbeat monitoring. T-Rehab closely monitors patients’ mental health as they engage in a personalized, semi-immersive Virtual Reality (VR) game on a desktop PC, using a webcam with MediaPipe to track their hand movements for interactive exercises, allowing the system to tailor treatment content for increased engagement and comfort. T-Rehab’s evaluation comprises two assessments: system performance and cognitive walkthroughs. The first evaluation focuses on system performance, assessing the tested game, middleware, and facial emotion monitoring to ensure hardware compatibility and effective support for AC, gaming, and tele-rehabilitation. The second evaluation uses cognitive walkthroughs to examine usability, identifying potential issues in emotion detection and tele-rehabilitation. Together, these evaluations provide insights into T-Rehab’s functionality, usability, and impact in supporting both physical rehabilitation and emotional well-being. The thorough integration of technology inside T-Rehab ensures a holistic approach to tele-rehabilitation, allowing patients to participate comfortably and efficiently from anywhere. This technique not only improves physical therapy outcomes but also promotes mental resilience, marking an important step advance in tele-rehabilitation practices.

Recent advances in immersive computing and artificial intelligence have enabled the development of adaptive, human-centered digital healthcare systems. This paper presents CareSync, an emotion-aware, data-driven architecture for personalized interventions and Multimedia Information Retrieval within the Healthcare Metaverse. Built on the SenseCare KM-EP platform, the CareSync framework integrates distributed healthcare services to orchestrate real-time, multi-source data processing, multimodal feature extraction, and semantic graph generation. This work advances T-Rehab, the platform’s metaverse-based telerehabilitation subsystem, by proposing a workflow for integrating multimedia information retrieval (MIR), physiological sensing, and markerless kinematic analytics within a metaverse-based healthcare environment, and by implementing this workflow as a functional real-time system. The implementation integrates camera-based physiological sensing, including POS-based remote photoplethysmography (rPPG) heart rate estimation with bandpass autocorrelation, chest-motion respiratory rate estimation with RIIV cross-validation, and RMSSD-based stress heuristics, with markerless biomechanical analysis. Using MediaPipe and Leap Motion tracking, the platform further assesses facial emotions, pain-related behavioral cues, and full-body kinematics, including joint angles, range of motion, movement symmetry, and repetition counting, enabling detailed real-time monitoring during immersive telerehabilitation sessions. For data indexing and retrieval, structured session outcome summaries and co-occurrence graphs are generated from telemetry and vitals, supporting both exact filter queries and graph-similarity ranking for sessions preserved via non-video 3D avatar replays. Actionable information is surfaced through real-time linear-trend and threshold-based alerts that monitor system-estimated indicators of participant stability, safety, and affect. The updated T-Rehab was evaluated on the (N = 26) participant cohort from the rPPG-10 benchmark dataset (27 subjects in total, with Subject 4 excluded). The evaluation compared five rPPG methods, GREEN, CHROM, POS, PCA, and ICA, using synchronized 30-s windows and the ECG reference. The results identified GREEN as the best-performing method, achieving the lowest mean absolute error (MAE) of 17.74 BPM. Furthermore, the participant-level evaluation and consistent 30-s windowing provide a transparent and reproducible basis for comparing non-contact heart-rate estimation methods. The results are intended as a methodological benchmark and should not be interpreted as demonstrating clinical accuracy.