Generative Artificial Intelligence is set to revolutionize healthcare delivery by transforming traditional patient care into a more personalized, efficient, and proactive process. Chatbots, serving as interactive conversational models, will probably drive this patient-centered transformation in healthcare. Through the provision of various services, including diagnosis, personalized lifestyle recommendations, dynamic scheduling of follow-ups, and mental health support, the objective is to substantially augment patient health outcomes, all the while mitigating the workload burden on healthcare providers. The life-critical nature of healthcare applications necessitates establishing a unified and comprehensive set of evaluation metrics for conversational models. Existing evaluation metrics proposed for various generic large language models (LLMs) demonstrate a lack of comprehension regarding medical and health concepts and their significance in promoting patients’ well-being. Moreover, these metrics neglect pivotal user-centered aspects, including trust-building, ethics, personalization, empathy, user comprehension, and emotional support. The purpose of this paper is to explore state-of-the-art LLM-based evaluation metrics that are specifically applicable to the assessment of interactive conversational models in healthcare. Subsequently, we present a comprehensive set of evaluation metrics designed to thoroughly assess the performance of healthcare chatbots from an end-user perspective. These metrics encompass an evaluation of language processing abilities, impact on real-world clinical tasks, and effectiveness in user-interactive conversations. Finally, we engage in a discussion concerning the challenges associated with defining and implementing these metrics, with particular emphasis on confounding factors such as the target audience, evaluation methods, and prompt techniques involved in the evaluation process.

Background: Photoplethysmography is a noninvasive and low-cost method to remotely and continuously track vital signs. The Oura Ring is a compact photoplethysmography-based smart ring, which has recently drawn attention to remote health monitoring and wellness applications. The ring is used to acquire nocturnal heart rate (HR) and HR variability (HRV) parameters ubiquitously. However, these parameters are highly susceptible to motion artifacts and environmental noise. Therefore, a validity assessment of the parameters is required in everyday settings. Objective: This study aims to evaluate the accuracy of HR and time domain and frequency domain HRV parameters collected by the Oura Ring against a medical grade chest electrocardiogram monitor. Methods: We conducted overnight home-based monitoring using an Oura Ring and a Shimmer3 electrocardiogram device. The nocturnal HR and HRV parameters of 35 healthy individuals were collected and assessed. We evaluated the parameters within 2 tests, that is, values collected from 5-minute recordings (ie, short-term HRV analysis) and the average values per night sleep. A linear regression method, the Pearson correlation coefficient, and the Bland–Altman plot were used to compare the measurements of the 2 devices. Results: Our findings showed low mean biases of the HR and HRV parameters collected by the Oura Ring in both the 5-minute and average-per-night tests. In the 5-minute test, the error variances of the parameters were different. The parameters provided by the Oura Ring dashboard (ie, HR and root mean square of successive differences [RMSSD]) showed relatively low error variance compared with the HRV parameters extracted from the normal interbeat interval signals. The Pearson correlation coefficient tests (P<.001) indicated that HR, RMSSD, average of normal heart beat intervals (AVNN), and percentage of successive normal beat-to-beat intervals that differ by more than 50 ms (pNN50) had high positive correlations with the baseline values; SD of normal beat-to-beat intervals (SDNN) and high frequency (HF) had moderate positive correlations, and low frequency (LF) and LF:HF ratio had low positive correlations. The HR, RMSSD, AVNN, and pNN50 had narrow 95% CIs; however, SDNN, LF, HF, and LF:HF ratio had relatively wider 95% CIs. In contrast, the average-per-night test showed that the HR, RMSSD, SDNN, AVNN, pNN50, LF, and HF had high positive relationships (P<.001), and the LF:HF ratio had a moderate positive relationship (P<.001). The average-per-night test also indicated considerably lower error variances than the 5-minute test for the parameters. Conclusions: The Oura Ring could accurately measure nocturnal HR and RMSSD in both the 5-minute and average-per-night tests. It provided acceptable nocturnal AVNN, pNN50, HF, and SDNN accuracy in the average-per-night test but not in the 5-minute test. In contrast, the LF and LF:HF ratio of the ring had high error rates in both tests.

Background: Assessment of sleep quality is essential to address poor sleep quality and understand changes. Owing to the advances in the Internet of Things and wearable technologies, sleep monitoring under free-living conditions has become feasible and practicable. Smart rings and smartwatches can be employed to perform mid- or long-term home-based sleep monitoring. However, the validity of such wearables should be investigated in terms of sleep parameters. Sleep validation studies are mostly limited to short-term laboratory tests; there is a need for a study to assess the sleep attributes of wearables in everyday settings, where users engage in their daily routines. Objective: This study aims to evaluate the sleep parameters of the Oura ring along with the Samsung Gear Sport watch in comparison with a medically approved actigraphy device in a midterm everyday setting, where users engage in their daily routines. Methods: We conducted home-based sleep monitoring in which the sleep parameters of 45 healthy individuals (23 women and 22 men) were tracked for 7 days. Total sleep time (TST), sleep efficiency (SE), and wake after sleep onset (WASO) of the ring and watch were assessed using paired t tests, Bland-Altman plots, and Pearson correlation. The parameters were also investigated considering the gender of the participants as a dependent variable. Results: We found significant correlations between the ring’s and actigraphy’s TST (r=0.86; P<.001), WASO (r=0.41; P<.001), and SE (r=0.47; P<.001). Comparing the watch with actigraphy showed a significant correlation in TST (r=0.59; P<.001). The mean differences in TST, WASO, and SE of the ring and actigraphy were within satisfactory ranges, although there were significant differences between the parameters (P<.001); TST and SE mean differences were also within satisfactory ranges for the watch, and the WASO was slightly higher than the range (31.27, SD 35.15). However, the mean differences of the parameters between the watch and actigraphy were considerably higher than those of the ring. The watch also showed a significant difference in TST (P<.001) between female and male groups. Conclusions: In a sample population of healthy adults, the sleep parameters of both the Oura ring and Samsung watch have acceptable mean differences and indicate significant correlations with actigraphy, but the ring outperforms the watch in terms of the nonstaging sleep parameters.

Smart wristbands enable the continuous monitoring of health parameters, for example, in maternity care. Understanding the feasibility and acceptability of these devices in an authentic context is essential. The aim of this study was to evaluate the feasibility of using a smart wristband to collect continuous activity, sleep and heart rate data from the beginning of the second trimester until one month postpartum. The feasibility of a smart wristband was tested prospectively through pregnancy in nulliparous women (n = 20). The outcomes measured were the wear time of the device and the participants’ experiences with the smart wristband. The data were collected from the wristbands, phone interviews, questionnaires, and electronic patient records. The quantitative data were analyzed with hierarchical linear mixed models for repeated measures, and qualitative data were analyzed using content analysis. Participants (n = 20) were recruited at a median of 12.9 weeks of gestation. They used the smart wristbands for an average of 182 days during the seven-month study period. The daily use of the devices was similar during the second (17.9 h, 95% CI 15.2 to 20.7) and third trimesters (16.7 h, 95% CI 13.8 to 19.5) but decreased during the postpartum period (14.4 h, 95% CI 11.4 to 17.4, p = 0.0079). Participants who could not wear smart wristbands at work used the device 300 min less per day than did those with no use limitations. Eight of the participants did not wear the devices or wore them only occasionally after giving birth. Nineteen participants reported that the smart wristband did not have any permanent effects on their behavior. Problems with charging and synchronizing the devices, perceiving the devices as uncomfortable, or viewing the data as unreliable, and the fear of scratching their babies with the devices were the main reasons for not using the smart wristbands. A smart wristband is a feasible tool for continuous monitoring during pregnancy. However, the daily use decreased after birth. The results of this study may support the planning of future studies and help with overcoming barriers related to the use of smart wristbands on pregnant women.