Xinwei Peng

Xinwei Peng

Auto-generated profile, based on publicly available information

Xinwei Peng

Shanghai Artificial Intelligence Laboratory, Shanghai, China

Research Focus & Metrics

Academic Career Length

5Years (2022-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
28 June 2024 in JMIR Medical Informatics
Verified
Journal Article
Open access
ZXZhongzhou Xiao
Zhongzhou Xiao
LLLU LU
LU LU
XPXinwei Peng
Xinwei Peng
JPJiali Pang
Jiali Pang
JDJinru Ding
Jinru Ding
SZShaoting Zhang
Shaoting Zhang
Background: Large language models (LLMs) have achieved great progress in natural language processing tasks and demonstrated the potential for use in clinical applications. Despite their capabilities, LLMs in the medical domain are prone to generating hallucinations (not fully reliable responses). Hallucinations in LLMs’ responses create substantial risks, potentially threatening patients’ physical safety. Thus, to perceive and prevent this safety risk, it is essential to evaluate LLMs in the medical domain and build a systematic evaluation. Objective: We developed a comprehensive evaluation system, MedGPTEval, composed of criteria, medical data sets in Chinese, and publicly available benchmarks. Methods: First, a set of evaluation criteria was designed based on a comprehensive literature review. Second, existing candidate criteria were optimized by using a Delphi method with 5 experts in medicine and engineering. Third, 3 clinical experts designed medical data sets to interact with LLMs. Finally, benchmarking experiments were conducted on the data sets. The responses generated by chatbots based on LLMs were recorded for blind evaluations by 5 licensed medical experts. The evaluation criteria that were obtained covered medical professional capabilities, social comprehensive capabilities, contextual capabilities, and computational robustness, with 16 detailed indicators. The medical data sets include 27 medical dialogues and 7 case reports in Chinese. Three chatbots were evaluated: ChatGPT by OpenAI; ERNIE Bot by Baidu, Inc; and Doctor PuJiang (Dr PJ) by Shanghai Artificial Intelligence Laboratory. Results: Dr PJ outperformed ChatGPT and ERNIE Bot in the multiple-turn medical dialogues and case report scenarios. Dr PJ also outperformed ChatGPT in the semantic consistency rate and complete error rate category, indicating better robustness. However, Dr PJ had slightly lower scores in medical professional capabilities compared with ChatGPT in the multiple-turn dialogue scenario. Conclusions: MedGPTEval provides comprehensive criteria to evaluate chatbots by LLMs in the medical domain, open-source data sets, and benchmarks assessing 3 LLMs. Experimental results demonstrate that Dr PJ outperforms ChatGPT and ERNIE Bot in social and professional contexts. Therefore, such an assessment system can be easily adopted by researchers in this community to augment an open-source data set.

31 October 2025 in JMIR AI
Verified
Journal Article
Open access
LJLuyi Jiang
Luyi Jiang
JCJiayuan Chen
Jiayuan Chen
LLLU LU
LU LU
XPXinwei Peng
Xinwei Peng
LLLihao Liu
Lihao Liu
ZXZhongzhou Xiao
Zhongzhou Xiao
The evaluation and improvement of medical large language models (LLMs) are critical for their real-world deployment, particularly in ensuring accuracy, safety, and ethical alignment. However, existing frameworks are inadequate for dissecting domain-specific error patterns or addressing cross-modal challenges. To address the limitations of current evaluation methods, this study aims to develop and apply a granular error taxonomy to systematically identify critical weaknesses in leading medical LLMs. The ultimate goal is to establish an actionable road map for enhancing their clinical robustness, safety, and overall trustworthiness. This study introduces a granular error taxonomy developed through a systematic analysis of 10 top-performing models on MedBench (specifically AntAngelMed, Citrus-2.0, INF-Med, WHU_Med, zhuomuniao-Med, TeleChat2, hunyuan-med, UNI-GPT, fusiontech-Med, and GPT-4). Incorrect responses were categorized into 8 distinct types: omissions, hallucination, format mismatch, causal reasoning deficiency, contextual inconsistency, unanswered, output error, and deficiency in medical language generation. Based on these findings, we propose a tiered optimization strategy spanning 4 levels, from prompt engineering and knowledge-augmented retrieval to hybrid neuro-symbolic architectures and causal reasoning frameworks. Based on a comprehensive error analysis of 42,766 question responses generated by the top 10 models, the evaluation using the 8 defined metrics revealed significant vulnerabilities in the leading models. Despite achieving a high accuracy of 0.86 in medical knowledge recall, analysis across the 8 error categories identified omissions as the most prevalent issue, exhibiting a staggering 96.3% omission rate in critical reasoning tasks. Furthermore, safety and ethics evaluations showed alarming inconsistency under option-shuffled conditions, with a low robustness score of 0.79. Our analysis uncovers systemic weaknesses in the models' ability to enforce knowledge boundaries and perform multistep reasoning. This work establishes an actionable road map for developing more clinically robust LLMs. By providing error-driven insights, it redefines evaluation paradigms, ultimately advancing the safety and trustworthiness of artificial intelligence in high-stakes medical environments and promoting its responsible application.

20 January 2023 in Nutrients
Verified
Journal Article
Open access
PDF file
XPXinwei Peng
Xinwei Peng
JZJingjing Zhu
Jingjing Zhu
HLHenry Lynn
Henry Lynn
XZXi Zhang
Xi Zhang
Background: There is limited research on the associations between serum nutritional biomarkers and mortality risk in patients with metabolic syndrome (MetS). Existing studies merely investigated the single-biomarker effect. Thus, this study aimed to investigate the combined effect of nutritional biomarker mixtures and mortality risk using the Bayesian kernel machine regression (BKMR) model in patients with MetS. Methods: We included the MetS patients, defined according to the 2018 Guideline on the Management of Blood Cholesterol from the National Health and Nutrition Examination Survey (NHANES) 2001–2006. A total of 20 serum nutritional biomarkers were measured and evaluated in this study. The Cox proportional hazard model and restricted cubic spline models were used to evaluate the individual linear and non-linear association of 20 nutritional biomarkers with mortality risk. Bayesian kernel machine regression (BKMR) was used to assess the associations between mixture of nutritional biomarkers and mortality risk. Results: A total of 1455 MetS patients had a median age of 50 years (range: 20–85). During a median of 17.1-year follow-up, 453 (24.72%) died: 146 (7.20%) caused by CVD and 87 (5.26%) by cancer. Non-linear and linear analyses indicated that, in total, eight individual biomarkers (α-carotene, β-carotene, bicarbonate, lutein/zeaxanthin, lycopene, potassium, protein, and vitamin A) were significantly associated with all-cause mortality (all p-values < 0.05). Results from BKMR showed an association between the low levels of the mixture of nutritional biomarkers and high risk of all-cause mortality with the estimated effects ranging from 0.04 to 0.14 (referent: medians). α-Carotene (PIP = 0.971) and potassium (PIP = 0.796) were the primary contributors to the combined effect of the biomarker mixture. The nutritional mixture levels were found to be negatively associated with the risk of cardiovascular disease (CVD) mortality and positively associated with the risk of cancer mortality. After it was stratified by nutrients, the mixture of vitamins showed a negative association with all-cause and CVD mortality, whereas the mixture of mineral-related biomarkers was positively associated with all-cause and cancer mortality. Conclusion: Our findings support the evidence that nutritional status was associated with long-term health outcomes in MetS patients. It is necessary for MetS patients to be concerned with certain nutritional status (i.e., vitamins and mineral elements).

22 December 2022 in Healthcare
Verified
Journal Article
Open access
PDF file
XPXinwei Peng
Xinwei Peng
JLJi-Bin Li
Ji-Bin Li
YWYuwan Wu
Yuwan Wu
HDHongji Dai
Hongji Dai
HLHenry Lynn
Henry Lynn
XZXi Zhang
Xi Zhang
Background: Prior studies on the relationship between bowel health and mortality have generally focused on the individual association of stool frequency or consistency with mortality but did not present a joint association. Therefore, we aimed to systematically evaluate the individual and joint associations of stool frequency and consistency with all-cause and cause-specific mortality in this study. Methods: A total of 14,574 participants from the National Health and Nutrition Examination Survey 2005–2010 were incorporated in this analysis. Survey sample-weighted Cox proportional hazards models adjusted for potential confounders were used to estimate hazard ratios (HRs) between bowel health measures and mortality risks. Results: During a median of 7.6 years of follow-up, 1502 deaths occurred, including 357 cancer deaths and 284 cardiovascular disease (CVD) deaths. The bowel habit of the most participants was 7 times/week (50.7%), and the most common type was “Like a sausage or snake, smooth and soft” (51.8%). Stool frequency displayed a parabolic relationship with all-cause mortality, and less than 7 times/week is a significant risk factor for mortality (HR for 1 time/week: 1.43, p-values = 0.04. HR for 6 times/week: 1.05, p-value = 0.03). Analyzing the joint association of stool frequency and consistency on mortality clarified the limitations of only inspecting the effects of either individual factor. Compared with 7 times/week of normal stool, infrequent soft stools at 4 times/week were associated with 1.78-, 2.42-, and 2.27-times higher risks of all-cause, cancer, and CVD mortality, respectively. Conclusion: Analyses of bowel health should consider the joint effects of stool frequency and stool consistency. Self-appraisal of stool frequency and consistency may be a simple but useful tool for informing about major chronic illnesses.

30 October 2024 in Journal of Medical Internet Research
Verified
Journal Article
Open access
LYLingrui Yang
Lingrui Yang
JPJiali Pang
Jiali Pang
SZSong Zuo
Song Zuo
JXJian Xu
Jian Xu
WJWei Jin
Wei Jin
XSXin Sun
Xin Sun
Background: Although new technologies have increased the efficiency and convenience of medical care, patients still struggle to identify specialized outpatient departments in Chinese tertiary hospitals due to a lack of medical knowledge. Objective: The objective of our study was to develop a precise and subdividable outpatient triage system to improve the experiences and convenience of patient care. Methods: We collected 395,790 electronic medical records (EMRs) and 500 medical dialogue groups. The EMRs were divided into 3 data sets to design and train the triage model (n=387,876, 98%) and test (n=3957, 1%) and validate (n=3957, 1%) it. The triage system was altered based on the current BERT (Bidirectional Encoder Representations from Transformers) framework and evaluated by recommendation accuracies in Xinhua Hospital using the cancellation rates in 2021 and 2022, from October 29 to December 5. Finally, a prospective observational study containing 306 samples was conducted to compare the system’s performance with that of triage nurses, which was evaluated by calculating precision, accuracy, recall of the top 3 recommended departments (recall@3), and time consumption. Results: With 3957 (1%) records each, the testing and validation data sets achieved an accuracy of 0.8945 and 0.8941, respectively. Implemented in Xinhua Hospital, our triage system could accurately recommend 79 subspecialty departments and reduce the number of registration cancellations from 16,037 (3.83%) of the total 418,714 to 15,338 (3.53%) of the total 434200 (P<.05). In comparison to the triage system, the performance of the triage nurses was more accurate (0.9803 vs 0.9153) and precise (0.9213 vs 0.9049) since the system could identify subspecialty departments, whereas triage nurses or even general physicians can only recommend main departments. In addition, our triage system significantly outperformed triage nurses in recall@3 (0.6230 vs 0.5266; P<.001) and time consumption (10.11 vs 14.33 seconds; P<.001). Conclusions: The triage system demonstrates high accuracy in outpatient triage of all departments and excels in subspecialty department recommendations, which could decrease the cancellation rate and time consumption. It also improves the efficiency and convenience of clinical care to fulfill better the usage of medical resources, expand hospital effectiveness, and improve patient satisfaction in Chinese tertiary hospitals.

1 March 2023 in Infection and Drug Resistance
Verified
Journal Article
Open access
PDF file
MWMiao Wang
Miao Wang
JSJin Shi
Jin Shi
KZKepei Zhang
Kepei Zhang
JHJie Hong
Jie Hong
XPXinwei Peng
Xinwei Peng
GMGuojiang Mei
Guojiang Mei
Purpose: This study aimed to evaluate the clinical efficacy of Longyizhengqi granule, a traditional Chinese medicine, in patients with mild COVID-19. Patients and Methods: We conducted a prospective study including mild COVID-19 participants conducted at Mobile Cabin Hospital in Shanghai, China. Participants were assigned to receive Longyizhengqi granule or conventional treatment. The primary outcome was the time for nucleic acid to turn negative and the secondary outcomes are hospital stay and changes in cycle threshold (Ct) values for N gene and Orf gene. Multilevel random-intercept model was performed to analyze the effects of treatment. Results: A total of 3243 patients were included in this study (Longyizhengqi granule 667 patients; conventional treatment 2576 patients). Age (43.5 vs 42.1, p< 0.01) and vaccination doses (not vaccinated: 15.8% vs 21.7%, 1 dose: 3.5% vs 2.9%, 2 doses: 27.9% vs 25.6%, 3 doses: 52.8% vs 49.8%. p< 0.01) show statistical difference between Conventional treatment group and LYZQ granules group. The use of Longyizhengqi granule could significantly reduce the time for nucleic acid to turn negative (14.2 days vs 10.7 days, p< 0.01), shorten hospital stay (12.5 days vs 9.9 days, p< 0.01), and increase the changes in Ct value for N gene (8.44 vs 10.33, p< 0.01) and Orf gene (7.31 vs 8.44, p< 0.01) to approximately 1.5. Moreover, the difference in the changes of Ct values on the 4th, 6th, 8th, and 10th days seem to increase between two groups. No serious adverse events were reported. Conclusion: Longyizhengqi granule might be a promising drug for the treatment of mild COVID-19, and it might be beneficial to effectively shorten the negative transition time of nucleic acid, the total days of hospitalization, and increase the changes of Ct values. Long-term randomized controlled trials with follow-up evaluations are required to confirm its long-term efficacy.