Background: Hyperkinetic movement disorders involve excessive, involuntary movements such as ataxia, chorea, dystonia, myoclonus, tics, and tremor. Recent advances in artificial intelligence (AI) allow investigators to integrate multimodal instrumented movement measurements and imaging techniques and to analyze these data together at scale. In this systematic review, we aim to characterize AI’s performance in diagnosing and quantitatively phenotyping these disorders. Methods: We searched PubMed and Embase using a semi-automated article-screening pipeline. Results: Fifty-five studies met the inclusion criteria (n = 11,946 subjects). Thirty-five studies used machine learning, sixteen used deep learning, and four used both. Thirty-eight studies reported disease diagnosis, twenty-three reported quantitative phenotyping, and six reported both. Diagnostic accuracy was reported in 36 of 38 and correlation coefficients in 10 of 23 studies. Kinematics (e.g., accelerometers and inertial measurement units) were the most used dataset. Diagnostic accuracy was reported in 36 studies and ranged from 56 to 100% compared to clinical diagnoses to differentiate them from healthy controls. The correlation coefficient was reported in 10 studies and ranged from 0.54 to 0.99 compared to clinical ratings for quantitative phenotyping. Five studies had an overall judgment of “low risk of bias” and three had external validation. Conclusion: There is a need to adopt AI-based research guidelines to minimize reporting heterogeneity and bolster clinical interpretability.

Objective: Large-language models (LLMs) can potentially revolutionize health care delivery and research, but risk propagating existing biases or introducing new ones. In epilepsy, social determinants of health are associated with disparities in care access, but their impact on seizure outcomes among those with access remains unclear. Here we (1) evaluated our validated, epilepsy-specific LLM for intrinsic bias, and (2) used LLM-extracted seizure outcomes to determine if different demographic groups have different seizure outcomes. Materials and Methods: We tested our LLM for differences and equivalences in prediction accuracy and confidence across demographic groups defined by race, ethnicity, sex, income, and health insurance, using manually annotated notes. Next, we used LLM-classified seizure freedom at each office visit to test for demographic outcome disparities, using univariable and multivariable analyses. Results: We analyzed 84 675 clinic visits from 25 612 unique patients seen at our epilepsy center. We found little evidence of bias in the prediction accuracy or confidence of outcome classifications across demographic groups. Multivariable analysis indicated worse seizure outcomes for female patients (OR 1.33, P ≤ .001), those with public insurance (OR 1.53, P ≤ .001), and those from lower-income zip codes (OR ≥1.22, P ≤ .007). Black patients had worse outcomes than White patients in univariable but not multivariable analysis (OR 1.03, P = .66). Conclusion: We found little evidence that our LLM was intrinsically biased against any demographic group. Seizure freedom extracted by LLM revealed disparities in seizure outcomes across several demographic groups. These findings quantify the critical need to reduce disparities in the care of people with epilepsy.

Objective: Large-language models (LLMs) in healthcare have the potential to propagate existing biases or introduce new ones. For people with epilepsy, social determinants of health are associated with disparities in access to care, but their impact on seizure outcomes among those with access to specialty care remains unclear. Here we (1) evaluated our validated, epilepsy-specific LLM for intrinsic bias, and (2) used LLM-extracted seizure outcomes to test the hypothesis that different demographic groups have different seizure outcomes. Methods: First, we tested our LLM for intrinsic bias in the form of differential performance in demographic groups by race, ethnicity, sex, income, and health insurance in manually annotated notes. Next, we used LLM-classified seizure freedom at each office visit to test for outcome disparities in the same demographic groups, using univariable and multivariable analyses. Results: We analyzed 84,675 clinic visits from 25,612 patients seen at our epilepsy center 2005-2022. We found no differences in the accuracy, or positive or negative class balance of outcome classifications across demographic groups. Multivariable analysis indicated worse seizure outcomes for female patients (OR 1.33, p = 3x10 ^-8 ), those with public insurance (OR 1.53, p = 2x10 ^-13 ), and those from lower-income zip codes (OR ≥ 1.22, p ≤ 6.6x10 ^-3 ). Black patients had worse outcomes than White patients in univariable but not multivariable analysis (OR 1.03, p = 0.66). Significance: We found no evidence that our LLM was intrinsically biased against any demographic group. Seizure freedom extracted by LLM revealed disparities in seizure outcomes across several demographic groups. These findings highlight the critical need to reduce disparities in the care of people with epilepsy. Key Points: We used large language models (LLMs) and natural language processing to extract seizure outcomes from clinical note text. We found no evidence of intrinsic bias in the LLM algorithm, in that it performed similarly across all demographic groups. Using LLM-extracted seizure outcomes, female sex, public insurance, and lower income zip- codes were associated with higher likelihood of seizures at each visit. Black race was associated with higher likelihood of seizures in univariable but not multivariable analysis. These findings highlight the critical need to reduce disparities in the care of people with epilepsy.

Objective: We have previously developed a natural language processing pipeline using clinical notes written by epilepsy specialists to extract seizure freedom, seizure frequency text, and date of last seizure text for patients with epilepsy. It is important to understand how our methods generalize to new care contexts. Materials and methods: We evaluated our pipeline on unseen notes from nonepilepsy-specialist neurologists and non-neurologists without any additional algorithm training. We tested the pipeline out-of-institution using epilepsy specialist notes from an outside medical center with only minor preprocessing adaptations. We examined reasons for discrepancies in performance in new contexts by measuring physical and semantic similarities between documents. Results: Our ability to classify patient seizure freedom decreased by at least 0.12 agreement when moving from epilepsy specialists to nonspecialists or other institutions. On notes from our institution, textual overlap between the extracted outcomes and the gold standard annotations attained from manual chart review decreased by at least 0.11 F1 when an answer existed but did not change when no answer existed; here our models generalized on notes from the outside institution, losing at most 0.02 agreement. We analyzed textual differences and found that syntactic and semantic differences in both clinically relevant sentences and surrounding contexts significantly influenced model performance. Discussion and conclusion: Model generalization performance decreased on notes from nonspecialists; out-of-institution generalization on epilepsy specialist notes required small changes to preprocessing but was especially good for seizure frequency text and date of last seizure text, opening opportunities for multicenter collaborations using these outcomes.