The current research evaluates user experience and preference when interacting with a patient-reported outcome measure (PROM) healthcare application displayed on a single tablet in comparison to interaction with the same application distributed across two tablets. We conducted a within-subject user study with 43 participants who engaged with and rated the usability of our system and participated in a post-experiment interview to collect subjective data. Our findings showed significantly higher usability and higher pragmatic quality ratings for the single tablet condition. However, some users attribute a higher level of presence to the avatar and prefer it to be placed on a second tablet.

The collection of patient-reported outcome measures (PROMs) is a key measurement tool for patient-centred care. At the same time, collecting these measures poses obstacles for many patients, leading to these groups being underrepresented in the data. We have therefore developed a multimodal, AI-driven assistance system to support patients in collecting these data. The interface of the system comprised a digital tablet containing the PROM questionnaire items and the assistant in three forms of embodiment: A virtual avatar, a physical avatar, and a voice-only agent. To evaluate the users’ experience and ratings of the system, two separate studies were implemented in two rehabilitation centers with 195 patients. A mixed within–between RCT was conducted at an outpatient clinic, where patients completed PROMs both with and without an assistant, and a between-subject design at an inpatient clinic comparing routine PC-based care with avatar- and robot-assisted PROM administration. Our results suggest a preference for the non-assisted tablet-only condition in Clinic A, whereas, in Clinic B, both agent conditions were preferred over routine care. We have further analyzed aspects such as trust and social presence in this study to gain a more thorough understanding of the users’ experience. Our analysis shows a higher trust rating for the voice-only assistant, whereas the robot, virtual avatar, and the voice-only conditions were perceived as more socially present. The impact of demographic factors and affinity for technology on the user ratings was also thoroughly studied. Our findings shed light on the role of agent embodiment in PROM assistance and contribute to the future design and evaluation of effective, engaging, and trustworthy systems for data collection in healthcare settings.

BACKGROUND Patient-reported outcome measure (PROM) completion is hindered by patient-level barriers—including motor, sensory, cognitive, and motivational constraints—that risk insufficient participation and non-response bias. While technology-enabled approaches such as multimodal speech assistance hold promise for reducing these barriers, assistance is a complex interaction: it can both alleviate and introduce barriers depending on how well it aligns with patients’ routines and needs. OBJECTIVE This qualitative study explores how patients perceive the advantages and disadvantages of AI-based speech assistance for PROM collection, focusing on how assistance functionalities interact with individual barriers and completion practices. METHODS We conducted semi-structured qualitative interviews with 96 psychosomatic and neurological rehabilitation outpatients, embedded in a pragmatic cross-randomised controlled trial. Participants completed PROMs with and without an AI-based speech assistance system offering speech output, speech input, and guidance by a socially interactive agent (SIA) that was physically, virtually, or voice-only embodied. The system was iteratively refined during data collection to address usability and performance issues. We included a broad sample to reflect real-world care settings, including patients without reported barriers. Using inductive content analysis (61 codes, grouped into 4 overarching and 9 subthemes), we examined perceived advantages and disadvantages of the three main assistance functionalities and multimodal interaction. Reporting followed the COREQ guideline. RESULTS The speech output function emerged as the most widely valued assistance feature, with many patients reporting improved concentration, question comprehension, and deeper engagement with item content. The social agent was described as making the interaction more engaging and less monotonous, by at the same time not evoking social pressure. Speech input was perceived as helpful by some, especially for those with motor impairments or a preference for verbal expression. However, each function also introduced challenges: speech output disrupted reading routines for some, the social agent was perceived as distracting or unnecessary by others, and speech input was criticised for recognition errors, inefficiency, and privacy concerns. CONCLUSIONS AI-based speech assistance for PROM collection offers significant potential to reduce barriers and enhance patient engagement, but its effectiveness depends on alignment with individual needs, preferences and routines. While speech output proved broadly beneficial, speech input and socially interactive agents require careful design to avoid introducing new barriers, particularly for marginalised groups. Configurable, modular assistance systems that adapt to diverse user preferences and impairments are essential for equitable implementation. Future research should focus on inclusive co-design and longitudinal studies to refine these technologies for real-world clinical use. CLINICALTRIAL German Clinical Trail Register-ID: DRKS00035213