Tremor is one of the common symptoms of Parkinson’s disease (PD). Thanks to the recent evolution of digital technologies, monitoring of PD patients’ hand movements employing contactless methods gained momentum. Objective: We aimed to quantitatively assess hand movements in patients suffering from PD using the artificial intelligence (AI)-based hand-tracking technologies of MediaPipe. Method: High-frame-rate videos and accelerometer data were recorded from 11 PD patients, two of whom showed classical Parkinsonian-type tremor. In the OFF-state and 30 Minutes after taking their standard oral medication (ON-state), video recordings were obtained. First, we investigated the frequency and amplitude relationship between the video and accelerometer data. Then, we focused on quantifying the effect of taking standard oral treatments. Results: The data extracted from the video correlated well with the accelerometer-based measurement system. Our video-based approach identified the tremor frequency with a small error rate (mean absolute error 0.229 (±0.174) Hz) and an amplitude with a high correlation. The frequency and amplitude of the hand movement before and after medication in PD patients undergoing medication differ. PD Patients experienced a decrease in the mean value for frequency from 2.012 (±1.385) Hz to 1.526 (±1.007) Hz and in the mean value for amplitude from 8.167 (±15.687) a.u. to 4.033 (±5.671) a.u. Conclusions: Our work achieved an automatic estimation of the movement frequency, including the tremor frequency with a low error rate, and to the best of our knowledge, this is the first paper that presents automated tremor analysis before/after medication in PD, in particular using high-frame-rate video data.

In recent years, significant research has been conducted on video-based human pose estimation (HPE). While monocular two-dimensional (2D) HPE has been shown to achieve high performance, monocular three-dimensional (3D) HPE poses a more challenging problem. However, since human motion happens in a 3D space, 3D HPE offers a more accurate representation of the human, granting increased usability for complex tasks like analysis of physical exercise. We propose a method based on MediaPipe Pose, 2D HPE on stereo cameras and a fusion algorithm without prior stereo calibration to reconstruct 3D poses, combining the advantages of high accuracy in 2D HPE with the increased usability of 3D coordinates. We evaluate this method on a self-recorded database focused on physical exercise to research what accuracy can be achieved and whether this accuracy is sufficient to recognize errors in exercise performance. We find that our method achieves significantly improved performance compared to monocular 3D HPE (median RMSE of $30.1$ compared to $56.3$, p-value below ${10}^{-6}$) and can show that the performance is sufficient for error recognition.

With the recent increase in interest in machine learning and computer vision, camera-based pose estimation has emerged as a promising new technology. One of the most popular libraries for camera-based pose estimation is MediaPipe Pose due to its computational efficiency, ease of use, and the fact that it is open-source. However, little work has been performed to establish how accurate the library is and whether it is suitable for usage in, for example, physical therapy. This paper aims to provide an initial assessment of this. We find that the pose estimation is highly dependent on the camera’s viewing angle as well as the performed exercise. While high accuracy can be achieved under optimal conditions, the accuracy quickly decreases when the conditions are less favourable.

Background: Many studies have shown a direct relationship between physical activity and health. It has also been shown that the average fitness level in Western societies is lower than recommended by the World Health Organization. One tool that can be used to increase physical activity for individual people is exergaming, that is, serious games that motivate players to do physical exercises. Objective: This scoping review of recent studies regarding exergame efficacy aims to evaluate which sensing modalities are used to assess exergame efficacy as well as motion quality. We also analyze how the collected motion sensing data is being leveraged with respect to exergame efficacy and motion quality assessment. Methods: We conducted 2 extensive and systematic searches of the ACM Digital Library and the PubMed database, as well as a single search of the IEEE Xplore database, all according to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement. Overall, 343 studies were assessed for eligibility by the following criteria: The study should be peer-reviewed; the year of publication should be between 2015 and 2023; the study should be available in English or German; the study evaluates the efficacy of at least 1 exergame; sensor data is recorded during the study and is used for evaluation; and the study is sufficiently described to extract information on the exergames, sensors, metrics, and results. Results: We found 67 eligible studies, which we analyzed with regard to sensor usage for both efficacy evaluation and motion analysis. Overall, heart rate (HR) was the most commonly used vital sign to evaluate efficacy (n=52), while the Microsoft Kinect was the most commonly used exergame sensor (n=26). The results of the analysis show that the sensors used in the exergames and the sensors used in the evaluation are, in most cases, mutually exclusive, with motion quality rarely being considered as a metric. Conclusions: The lack of motion quality assessment is identified as a problem both for the studies and the exergames themselves since incorrectly executed motions can reduce an exergame’s effectiveness and increase the risk of injury. Here we propose how to use the same sensors both as input for the exergame and to assess motion quality by presenting recent developments in motion recognition and sensing.

Objective: Cardiovascular diseases (CVDs) account for a high fatality rate worldwide. Heart murmurs can be detected from phonocardiograms (PCGs) and may indicate CVDs. Still, they are often overlooked as their detection and correct clinical interpretation require expert skills. In this work, we aim to predict the presence of murmurs and clinical outcomes from multiple PCG recordings employing an explainable multitask model. Approach: Our approach consists of a two-stage multitask model. In the first stage, we predict the murmur presence in single PCGs using a multiple instance learning (MIL) framework. MIL also allows us to derive sample-wise classifications (i.e. murmur locations) while only needing one annotation per recording (“weak label”) during training. In the second stage, we fuse explainable hand-crafted features with features from a pooling-based artificial neural network (PANN) derived from the MIL framework. Finally, we predict the presence of murmurs and the clinical outcome for a single patient based on multiple recordings using a simple feed-forward neural network. Main results: We show qualitatively and quantitatively that the MIL approach yields useful features and can be used to detect murmurs on multiple time instances and may thus guide a practitioner through PCGs. We analyze the second stage of the model in terms of murmur classification and clinical outcome. We achieved a weighted accuracy of 0.714 and an outcome cost of 13612 when using the PANN model and demographic features on the CirCor dataset (hidden test set of the George B. Moody PhysioNet challenge 2022, team “Heart2Beat”, rank 12 / 40). Significance: To the best of our knowledge, we are the first to demonstrate the usefulness of MIL in PCG classification. Also, we showcase how the explainability of the model can be analyzed quantitatively, thus avoiding confirmation bias inherent to many post-hoc methods. Finally, our overall results demonstrate the merit of employing MIL combined with handcrafted features for the generation of explainable features as well as for a competitive classification performance.

Objective Cardiovascular diseases (CVDs) account for a high fatality rate worldwide. Heart murmurs can be detected from phonocardiograms (PCGs) and may indicate CVDs. Still they are often overlooked as their detection and correct clinical interpretation requires expert skills. In this work, we aim to predict the presence of murmurs and clinical outcome from multiple PCG recordings employing an explainable multitask model. Approach Our approach consists of a two-stage multitask model. In the first stage, we predict the murmur presence in single PCGs using a multiple instance learning (MIL) framework. MIL also allows us to derive sample-wise classifications (i.e. murmur locations) while only needing one annotation per recording (“weak label”) during training. In the second stage, we fuse explainable hand-crafted features with features from a pooling-based artificial neural network (PANN) derived from the MIL framework. Finally, we predict the presence of murmurs as well as the clinical outcome for a single patient based on multiple recordings using a simple feed-forward neural network. Main results We show qualitatively and quantitatively that the MIL approach yields useful features and can be used to detect murmurs on multiple time instances and may thus guide a practitioner through PCGs. We analyze the second stage of the model in terms of murmur classification and clinical outcome. We achieved a weighted accuracy of 0.714 and an outcome cost of 13612 when using the PANN model and demographic features on the CirCor dataset (hidden testset of the George B. Moody PhysioNet challenge 2022, team “Heart2Beat”, rank 12 / 40). Significance To the best of our knowledge, we are the first to demonstrate the usefulness of MIL in PCG classification. Also, we showcase how the explainability of the model can be analyzed quantitatively, thus avoiding confirmation bias inherent to many post-hoc methods. Finally, our overall results demonstrate the merit of employing MIL combined with handcrafted features for the generation of explainable features as well as for a competitive classification performance.