21 November 2024 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Large language models (LLMs) have the potential to transform digital healthcare, as evidenced by recent advances in LLM-based virtual doctors. However, current approaches rely on patient's subjective descriptions of symptoms, causing increased misdiagnosis. Recognizing the value of daily data from smart devices, we introduce a novel LLM-based multi-turn consultation virtual doctor system, DrHouse, which incorporates three significant contributions: 1) It utilizes sensor data from smart devices in the diagnosis process, enhancing accuracy and reliability. 2) DrHouse leverages continuously updating medical knowledge bases to ensure its model remains at diagnostic standard's forefront. 3) DrHouse introduces a novel diagnostic algorithm that concurrently evaluates potential diseases and their likelihood, facilitating more nuanced and informed medical assessments. Through multi-turn interactions, DrHouse determines the next steps, such as accessing daily data from smart devices or requesting in-lab tests, and progressively refines its diagnoses. Evaluations on three public datasets and our self-collected datasets show that DrHouse can achieve up to an 31.5% increase in diagnosis accuracy over the state-of-the-art baselines. The results of a 32-participant user study show that 75% medical experts and 91.7% test subjects are willing to use DrHouse.
1 July 2021 in Biomedical Signal Processing and Control
3 March 2025 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Social interactions are fundamental to human life. The recent emergence of large language models (LLMs)-based virtual assistants has demonstrated their potential to revolutionize human interactions and lifestyles. However, existing assistive systems mainly provide reactive services to individual users, rather than offering in-situ assistance during live social interactions with conversational partners. In this study, we introduce SocialMind, the first LLM-based proactive AR social assistive system that provides users with in-situ social assistance. SocialMind employs human-like perception leveraging multi-modal sensors to extract both verbal and nonverbal cues, social factors, and implicit personas, incorporating these social cues into LLM reasoning for social suggestion generation. Additionally, SocialMind employs a multi-tier collaborative generation strategy and proactive update mechanism to display social suggestions on Augmented Reality (AR) glasses, ensuring that suggestions are timely provided to users without disrupting the natural flow of conversation. Evaluations on three public datasets and a user study with 20 participants show that SocialMind achieves 38.3% higher engagement compared to baselines, and 95% of participants are willing to use SocialMind in their live social interactions.
4 June 2020 in IEEE Access
Monitoring single-channel EEG is a promising home-based approach for insomnia identification. Currently, many automatic sleep stage scoring approaches based on single-channel EEG have been developed, whereas few studies research on automatic insomnia identification based on single-channel EEG labelled with sleep stage annotations. In this paper, we propose a one-dimensional convolutional neural network (1D-CNN) model for automatic insomnia identification based on single-channel EEG labelled with sleep stage annotations, and further investigate the identification performance based on different sleep stages EEG epochs. Single-channel EEG on 9 insomnia patients and 9 healthy subjects was used in this study. We constructed 4 subdatasets from EEG epochs based on the sleep stage annotations: All sleep stage dataset (ALL-DS), REM sleep stage dataset (REM-DS), light sleep stage dataset (LSS-DS), and SWS sleep stage dataset (SWS-DS). Subsequently, 4 subdatasets were fed into our 1D-CNN. We conducted experiments under intra-patient and inter-patient paradigms, respectively. Our experiments demonstrated that our 1D-CNN leveraging 3 subdatasets composed of REM, LSS and SWS epochs, respectively, achieved higher average accuracies in comparison with baseline methods under both intra-patient and inter-patient paradigms. The experimental results also indicated that amongst all the sleep stages, 1D-CNN leveraging REM and SWS epochs exhibited the best insomnia identification average accuracies in intra-patient paradigm, which are 98.98% and 99.16% respectively, whereas no statistically significant difference was found in inter-patient paradigm. For automatic insomnia identification based on single-channel EEG labelled with sleep stage annotations, 1D-CNN model introduced in this paper could achieve superior performance than traditional methods.