Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). Background: Individuals living with long-term physical health conditions frequently experience co-occurring mental health problems. This comorbidity has a significant impact on individual’s levels of emotional distress, health outcomes and associated healthcare utilization. As healthcare services struggle to meet demand and care increasingly moves to the community, digital tools are being promoted to support patients to self-manage their health. One such technology is the autonomous virtual agent. Objective: To co-design the content, functionality and interface modalities of an autonomous virtual agent to support self-management for patients with an exemplar long-term condition (Chronic Pulmonary Obstructive Disease, COPD), and then to assess the acceptability and system content. Methods: We conducted two co-design workshops and a proof-of-concept implementation of an autonomous virtual agent with natural language processing capabilities. This implementation formed the basis for video-based scenario testing of acceptability with adults with a diagnosis of COPD and health professionals involved in their care. Results: Adults (n = 6) with a diagnosis of COPD and health professionals (n = 5) specified four priority self-management scenarios for which they would like to receive support: at time of diagnosis (information provision); during acute exacerbations (crisis support); during periods of low mood (emotional support); and for general self-management (motivation). Based on the scenario testing, 12 additional adults with COPD felt the system was both acceptable and engaging, particularly with respect to internet of things capabilities. They felt the system would be particularly useful for individuals living alone. Conclusions: Patients did not explicitly separate out mental and physical health needs, although the content they developed for Avachat had a clear psychological approach. Supported self-management delivered via an autonomous virtual agent was acceptable to the participants. A co-design process has allowed the research team to identify key design principles, content and functionality to underpin an autonomous agent for delivering self-management support to older adults living with COPD and potentially other long term conditions.

Neurodegenerative diseases causing dementia are known to affect a person’s speech and language. Part of the expert assessment in memory clinics therefore routinely focuses on detecting such features. The current outpatient procedures examining patients’ verbal and interactional abilities mainly focus on verbal recall, word fluency, and comprehension. By capturing neurodegeneration-associated characteristics in a person’s voice, the incorporation of novel methods based on the automatic analysis of speech signals may give us more information about a person’s ability to interact which could contribute to the diagnostic process. In this proof-of-principle study, we demonstrate that purely acoustic features, extracted from recordings of patients’ answers to a neurologist’s questions in a specialist memory clinic can support the initial distinction between patients presenting with cognitive concerns attributable to progressive neurodegenerative disorders (ND) or Functional Memory Disorder (FMD, i.e., subjective memory concerns unassociated with objective cognitive deficits or a risk of progression). The study involved 15 FMD and 15 ND patients where a total of 51 acoustic features were extracted from the recordings. Feature selection was used to identify the most discriminating features which were then used to train five different machine learning classifiers to differentiate between the FMD/ND classes, achieving a mean classification accuracy of 96.2%. The discriminative power of purely acoustic approaches could be integrated into diagnostic pathways for patients presenting with memory concerns and are computationally less demanding than methods focusing on linguistic elements of speech and language that require automatic speech recognition and understanding.

The Dysarthric Expressed Emotional Database (DEED) is a novel, parallel multimodal (audio-visual) database of dysarthric and typical emotional speech in British English which is a first of its kind. It is an induced (elicited) emotional database that includes speech recorded in the six basic emotions: “happiness”, “sadness”, “anger”, “surprise”, “fear”, and “disgust”. A “neutral” state has also been recorded as a baseline condition. The dysarthric speech part includes recordings from 4 speakers: one female speaker with dysarthria due to cerebral palsy and 3 speakers with dysarthria due to Parkinson’s disease (2 female and 1 male). The typical speech part includes recordings from 21 typical speakers (9 female and 12 male). This paper describes the collection of the database, covering its design, development, technical information related to the data capture, and description of the data files and presents the validation methodology. The database was validated subjectively (human performance) and objectively (automatic recognition). The achieved results demonstrated that this database will be a valuable resource for understanding emotion communication by people with dysarthria and useful in the research field of dysarthric emotion classification. The database is freely available for research purposes under a Creative Commons licence at: https://sites.google.com/sheffield.ac.uk/deed

Dementia poses a significant global challenge, with profound personal, societal, and economic impacts. Although it is incurable, early detection is crucial for ensuring appropriate care and support. Dementia can impair a person's speech and language abilities, and studies have demonstrated promising results in using spoken language for automatic dementia detection. Recently, deep learning-based self-supervised learning models, such as wav2vec2.0 (w2v) and BERT, have shown success in extracting acoustic and linguistic information. However, most studies have relied on single datasets and relatively straightforward methods for extracting and combining acoustic and linguistic modalities. This paper presents an in-depth exploration of the application of SSL models in this context by proposing the Two-Step Attention-based Feature Combination Cross-attention system (TSAC-ATT) for speech-based dementia detection. The contributions of this paper are as follows: i) we explore and analyse acoustic and linguistic feature extraction pipelines using SSL models, including the proposed TSAC framework to create high-performing acoustic features from w2v's contextual layers; ii) we demonstrate that these features, when fused using cross-attention, outperform various feature combination approaches; iii) all experimental work is conducted on two publicly available datasets (DementiaBank and ADReSS), as well as the IVA dataset collected by the Royal Hallamshire Hospital, which includes recordings of the standard Cookie Theft task. We present state-of-the-art results, highlighting that acoustic-only features based on the w2v model can achieve very high performance across multiple datasets. Furthermore, we show that the upstream performance of the automatic speech recognition module does not always predict downstream classification performance.

Previous work on interactions in the memory clinic has shown that conversation analysis can be used to differentiate neurodegenerative dementia from functional memory disorder. Based on this work, a screening system was developed that uses a computerised ‘talking head’ (intelligent virtual agent) and a combination of automatic speech recognition and conversation analysis-informed programming. This system can reliably differentiate patients with functional memory disorder from those with neurodegenerative dementia by analysing the way they respond to questions from either a human doctor or the intelligent virtual agent. However, much of this computerised analysis has relied on simplistic, nonlinguistic phonetic features such as the length of pauses between talk by the two parties. To gain confidence in automation of the stratification procedure, this paper investigates whether the patients’ responses to questions asked by the intelligent virtual agent are qualitatively similar to those given in response to a doctor. All the participants in this study have a clear functional memory disorder or neurodegenerative dementia diagnosis. Analyses of patients’ responses to the intelligent virtual agent showed similar, diagnostically relevant sequential features to those found in responses to doctors’ questions. However, since the intelligent virtual agent’s questions are invariant, its use results in more consistent responses across people – regardless of diagnosis – which facilitates automatic speech recognition and makes it easier for a machine to learn patterns. Our analysis also shows why doctors do not always ask the same question in the exact same way to different patients. This sensitivity and adaptation to nuances of conversation may be interactionally helpful; for instance, altering a question may make it easier for patients to understand. While we demonstrate that some of what is said in such interactions is bound to be constructed collaboratively between doctor and patient, doctors could consider ensuring that certain, particularly important and/or relevant questions are asked in as invariant a form as possible to be better able to identify diagnostically relevant differences in patients’ responses.

The common causes of Transient Loss of Consciousness (TLOC) are syncope, epilepsy, and functional/dissociative seizures (FDS). Simple, questionnaire-based decision-making tools for non-specialists who may have to deal with TLOC (such as clinicians working in primary or emergency care) reliably differentiate between patients who have experienced syncope and those who have had one or more seizures but are more limited in their ability to differentiate between epileptic seizures and FDS. Previous conversation analysis research has demonstrated that qualitative expert analysis of how people talk to clinicians about their seizures can help distinguish between these two TLOC causes. This paper investigates whether automated language analysis - using semantic categories measured by the Linguistic Inquiry and Word Count (LIWC) toolkit - can contribute to the distinction between epilepsy and FDS. Using patient-only talk manually transcribed from recordings of 58 routine doctor-patient clinic interactions, we compared the word frequencies for 21 semantic categories and explored the predictive performance of these categories using 5 different machine learning algorithms. Machine learning algorithms trained using the chosen semantic categories and leave-one-out cross-validation were able to predict the diagnosis with an accuracy of up to 81%. The results of this proof of principle study suggest that the analysis of semantic variables in seizure descriptions could improve clinical decision tools for patients presenting with TLOC.