Lulu Li

Lulu Li

Auto-generated profile, based on publicly available information

Lulu Li

Department of Pharmacy, Fuwai Central China Cardiovascular Hospital, 1 Fuwai Road, Zhengdong New District, Zhengzhou, China, 86 18538298379

Research Focus & Metrics

Academic Career Length

2Years (2025-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
24 July 2025 in JMIR Medical Informatics
Verified
Journal Article
Open access
LLLulu Li
Lulu Li
PDPengqiang Du
Pengqiang Du
XHXiaojing Huang
Xiaojing Huang
HZHongwei Zhao
Hongwei Zhao
MNMing Ni
Ming Ni
AWAifeng Wang
Aifeng Wang
Background: Generative artificial intelligence (AI) systems are increasingly deployed in clinical pharmacy; yet, systematic evaluation of their efficacy, limitations, and risks across diverse practice scenarios remains limited. Objective: This study aims to quantitatively evaluate and compare the performance of 8 mainstream generative AI systems across 4 core clinical pharmacy scenarios—medication consultation, medication education, prescription review, and case analysis with pharmaceutical care—using a multidimensional framework. Methods: Forty-eight clinically validated questions were selected via stratified sampling from real-world sources (eg, hospital consultations, clinical case banks, and national pharmacist training databases). Three researchers simultaneously tested 8 different generative AI systems (ERNIE Bot, Doubao, Kimi, Qwen, GPT-4o, Gemini-1.5-Pro, Claude-3.5-Sonnet, and DeepSeek-R1) using standardized prompts within a single day (February 20, 2025). A double-blind scoring design was used, with 6 experienced clinical pharmacists (≥5 years experience) evaluating the AI responses across 6 dimensions: accuracy, rigor, applicability, logical coherence, conciseness, and universality, scored 0‐10 per predefined criteria (eg, −3 for inaccuracy and −2 for incomplete rigor). Statistical analysis used one-way ANOVA with Tukey Honestly Significant Difference (HSD) post hoc testing and intraclass correlation coefficients (ICC) for interrater reliability (2-way random model). Qualitative thematic analysis identified recurrent errors and limitations. Results: DeepSeek-R1 (DeepSeek) achieved the highest overall performance (mean composite score: medication consultation 9.4, SD 1.0; case analysis 9.3, SD 1.0), significantly outperforming others in complex tasks (P<.05). Critical limitations were observed across models, including high-risk decision errors—75% omitted critical contraindications (eg, ethambutol in optic neuritis) and a lack of localization—90% erroneously recommended macrolides for drug-resistant Mycoplasma pneumoniae (China’s high-resistance setting), while only DeepSeek-R1 aligned with updated American Academy of Pediatrics (AAP) guidelines for pediatric doxycycline. Complex reasoning deficits: only Claude-3.5-Sonnet detected a gender-diagnosis contradiction (prostatic hyperplasia in female); no model identified diazepam’s 7-day prescription limit. Interrater consistency was lowest for conciseness in case analysis (ICC=0.70), reflecting evaluator disagreement on complex outputs. ERNIE Bot (Baidu) consistently underperformed (case analysis: 6.8, SD 1.5; P<.001 vs DeepSeek-R1). Conclusions: While generative AI shows promise as a pharmacist assistance tool, significant limitations—including high-risk errors (eg, contraindication omissions), inadequate localization, and complex reasoning gaps—preclude autonomous clinical decision-making. Performance stratification highlights DeepSeek-R1’s current advantage, but all systems require optimization in dynamic knowledge updating, complex scenario reasoning, and output interpretability. Future deployment must prioritize human oversight (human-AI co-review), ethical safeguards, and continuous evaluation frameworks.

16 April 2025 in JMIR Preprints
Verified
Preprint
Open access
LLLulu Li
Lulu Li
AWAifeng Wang
Aifeng Wang
PDPengqiang Du
Pengqiang Du
XHXiaojing Huang
Xiaojing Huang
HZHongwei Zhao
Hongwei Zhao
MYMeng Yan
Meng Yan
BACKGROUND In recent years, the implementation of artificial intelligence (AI) in health care is progressively transforming medical fields.However, there remains a gap between technological potential and practical application, necessitating the establishment of a scientific evaluation system.Despite some existing research beginning to conduct clinical application assessments of generative AI dialogue systems, these efforts are largely limited to testing individual models on single tasks, lacking horizontal comparative analysis across multiple dialogue models and validation of continuous decision chains in real clinical scenarios.As generative artificial intelligence (AI) systems play an increasingly extensive role in the field of Medicine and Pharmacy, we need more research to explore this area. OBJECTIVE To systematically evaluate and compare the performance of eight mainstream generative AI systems, both domestic and international, across four core clinical pharmacy practice scenarios: medication consultation, medication education, prescription review, and case analysis with pharmaceutical care. This study aims to quantitatively assess their capabilities in addressing common clinical pharmacy practice problems. METHODS Assessment questions were systematically extracted from medication consultation clinic records, real clinical cases, and clinical pharmacist standardized training examination databases. Three researchers tested eight different generative AI systems on the same day using standardized "inquiry prompts." A double-blind scoring design was employed, with six experienced clinical pharmacists with extensive clinical backgrounds evaluating the AI responses on a 0-10 scale across six dimensions: accuracy, rigor, applicability, logical coherence, conciseness, and universality. Statistical analysis used one-way analysis of variance (ANOVA) to compare score differences between systems, with multiple comparison tests for significant results, and intraclass correlation coefficient (ICC) calculations to assess inter-rater consistency.Systematic descriptive evaluations of the AI-generated responses were also conducted. RESULTS DeepSeek-R1 demonstrated the best overall performance across all four task categories. Qwen, GPT-4o, Claude-3.5-Sonnet, and Gemini-1.5-Pro performed slightly inferior to DeepSeek-R1. Doubao and Kimi showed inconsistent performance, while ERNIE Bot performed the poorest. Comprehensive evaluation indicated that responses from existing generative AI systems still have certain limitations and should be used as clinical reference tools rather than independent clinical decision-making bases. Inter-rater consistency analysis showed good agreement (ICC>0.75) in evaluating medication consultation, medication education, and prescription review. However, the lowest consistency level (ICC=0.70) was observed in assessing the conciseness of case analysis and pharmaceutical care, reflecting significant cognitive differences among raters regarding evaluation standards for these complex issues. CONCLUSIONS The DeepSeek-R1 model demonstrates significant potential as a supportive decision-making tool in clinical pharmacy practice. However, overall, the current generative AI systems still require systematic improvement and refinement in their ability to handle multidimensional complex clinical pharmacy problems. CLINICALTRIAL none