Background: Large language models (LLMs) have achieved great progress in natural language processing tasks and demonstrated the potential for use in clinical applications. Despite their capabilities, LLMs in the medical domain are prone to generating hallucinations (not fully reliable responses). Hallucinations in LLMs’ responses create substantial risks, potentially threatening patients’ physical safety. Thus, to perceive and prevent this safety risk, it is essential to evaluate LLMs in the medical domain and build a systematic evaluation. Objective: We developed a comprehensive evaluation system, MedGPTEval, composed of criteria, medical data sets in Chinese, and publicly available benchmarks. Methods: First, a set of evaluation criteria was designed based on a comprehensive literature review. Second, existing candidate criteria were optimized by using a Delphi method with 5 experts in medicine and engineering. Third, 3 clinical experts designed medical data sets to interact with LLMs. Finally, benchmarking experiments were conducted on the data sets. The responses generated by chatbots based on LLMs were recorded for blind evaluations by 5 licensed medical experts. The evaluation criteria that were obtained covered medical professional capabilities, social comprehensive capabilities, contextual capabilities, and computational robustness, with 16 detailed indicators. The medical data sets include 27 medical dialogues and 7 case reports in Chinese. Three chatbots were evaluated: ChatGPT by OpenAI; ERNIE Bot by Baidu, Inc; and Doctor PuJiang (Dr PJ) by Shanghai Artificial Intelligence Laboratory. Results: Dr PJ outperformed ChatGPT and ERNIE Bot in the multiple-turn medical dialogues and case report scenarios. Dr PJ also outperformed ChatGPT in the semantic consistency rate and complete error rate category, indicating better robustness. However, Dr PJ had slightly lower scores in medical professional capabilities compared with ChatGPT in the multiple-turn dialogue scenario. Conclusions: MedGPTEval provides comprehensive criteria to evaluate chatbots by LLMs in the medical domain, open-source data sets, and benchmarks assessing 3 LLMs. Experimental results demonstrate that Dr PJ outperforms ChatGPT and ERNIE Bot in social and professional contexts. Therefore, such an assessment system can be easily adopted by researchers in this community to augment an open-source data set.

The evaluation and improvement of medical large language models (LLMs) are critical for their real-world deployment, particularly in ensuring accuracy, safety, and ethical alignment. However, existing frameworks are inadequate for dissecting domain-specific error patterns or addressing cross-modal challenges. To address the limitations of current evaluation methods, this study aims to develop and apply a granular error taxonomy to systematically identify critical weaknesses in leading medical LLMs. The ultimate goal is to establish an actionable road map for enhancing their clinical robustness, safety, and overall trustworthiness. This study introduces a granular error taxonomy developed through a systematic analysis of 10 top-performing models on MedBench (specifically AntAngelMed, Citrus-2.0, INF-Med, WHU_Med, zhuomuniao-Med, TeleChat2, hunyuan-med, UNI-GPT, fusiontech-Med, and GPT-4). Incorrect responses were categorized into 8 distinct types: omissions, hallucination, format mismatch, causal reasoning deficiency, contextual inconsistency, unanswered, output error, and deficiency in medical language generation. Based on these findings, we propose a tiered optimization strategy spanning 4 levels, from prompt engineering and knowledge-augmented retrieval to hybrid neuro-symbolic architectures and causal reasoning frameworks. Based on a comprehensive error analysis of 42,766 question responses generated by the top 10 models, the evaluation using the 8 defined metrics revealed significant vulnerabilities in the leading models. Despite achieving a high accuracy of 0.86 in medical knowledge recall, analysis across the 8 error categories identified omissions as the most prevalent issue, exhibiting a staggering 96.3% omission rate in critical reasoning tasks. Furthermore, safety and ethics evaluations showed alarming inconsistency under option-shuffled conditions, with a low robustness score of 0.79. Our analysis uncovers systemic weaknesses in the models' ability to enforce knowledge boundaries and perform multistep reasoning. This work establishes an actionable road map for developing more clinically robust LLMs. By providing error-driven insights, it redefines evaluation paradigms, ultimately advancing the safety and trustworthiness of artificial intelligence in high-stakes medical environments and promoting its responsible application.

This investigation was conceived to engineer and appraise a pioneering clinical nomogram, crafted to bridge the extant chasm in literature regarding the postoperative risk stratification for deep vein thrombosis (DVT) in the aftermath of lower extremity orthopedic procedures. This novel tool offers a sophisticated and discerning algorithm for risk prediction, heretofore unmet by existing methodologies. In this retrospective observational study, clinical records of hospitalized patients who underwent lower extremity orthopedic surgery were collected at the Wuxi TCM Hospital Affiliated to the Nanjing University of Chinese Medicine between Jan 2017 and Oct 2019. The univariate and multivariate analysis with the backward stepwise method was applied to select features for the predictive nomogram. The performance of the nomogram was evaluated with respect to its discriminant capability, calibration ability, and clinical utility. A total of 5773 in-hospital patients were eligible for the study, with the incidence of deep vein thrombosis being approximately 1 % in this population. Among 31 variables included, 5 of them were identified to be the predictive features in the nomogram, including age, mean corpuscular hemoglobin concentration (MCHC), D-dimer, platelet distribution width (PDW), and thrombin time (TT). The area under the receiver operating characteristic (ROC) curve in the training and validation cohort was 85.9 % (95%CI: 79.96 %-90.04 %) and 85.7 % (95%CI: 78.96 %-90.69 %), respectively. Both the calibration curves and decision curve analysis demonstrated the overall satisfactory performance of the model. Our groundbreaking nomogram is distinguished by its unparalleled accuracy in discriminative and calibrating functions, complemented by its tangible clinical applicability. This innovative instrument is set to empower clinicians with a robust framework for the accurate forecasting of postoperative DVT, thus facilitating the crafting of bespoke and prompt therapeutic strategies, aligning with the rigorous standards upheld by the most esteemed biomedical journals.

Background: Considering the high incidence of medical privacy disclosure, it is of vital importance to study doctors’ privacy protection behavior and its influencing factors. Objective: We aim to develop a scale for doctors’ protection of patients’ privacy in Chinese public medical institutions, following construction of a theoretical model framework through grounded theory, and subsequently to validate the scale to measure this protection behavior. Methods: Combined with the theoretical paradigm of protection motivation theory (PMT) and semistructured interview data, the grounded theory research method, followed by the Delphi expert and group discussion methods, a theoretical framework and initial scale for doctors in Chinese public medical institutions to protect patients' privacy was formed. The adjusted scale was collected online using a WeChat electronic survey measured using a 5-point Likert scale. Exploratory and confirmatory factor analysis (EFA and CFA) and tests to analyze reliability and validity were performed on the sample data. SPSS 19.0 and Amos 26.0 statistical analysis software were used for EFA and CFA of the sample data, respectively. Results: According to the internal logic of PMT, we developed a novel theoretical framework of a “storyline,” which was a process from being unaware of patients' privacy to having privacy protection behavior, that affected doctors' cognitive intermediary and changed the development of doctors' awareness, finally affecting actual privacy protection behavior in Chinese public medical institutions. Ultimately, we created a scale to measure 18 variables in the theoretical model, comprising 63 measurement items, with a total of 208 doctors participating in the scaling survey, who were predominantly educated to the master’s degree level (n=151, 72.6%). The department distribution was relatively balanced. Prior to EFA, the Kaiser-Meyer-Olkin (KMO) value was 0.702, indicating that the study was suitable for factor analysis. The minimum value of Cronbach α for each study variable was .754, which met the internal consistency requirements of the scale. The standard factor loading value of each potential measurement item in CFA had scores greater than 0.5, which signified that all the items in the scale could effectively converge to the corresponding potential variables. Conclusions: The theoretical framework and scale to assess doctors' patient protection behavior in public medical institutions in China fills a significant gap in the literature and can be used to further the current knowledge of physicians’ thought processes and adoption decisions.

UNSTRUCTURED The evaluation and improvement of medical large language models (LLMs) are critical for their real-world deployment, particularly in ensuring accuracy, safety, and ethical alignment. Existing frameworks inadequately dissect domain-specific error patterns or address cross-modal challenges. This study introduces a granular error taxonomy through systematic analysis of top 10 models on MedBench, categorizing incorrect responses into eight types: Omissions, Hallucination, Format Mismatch, Causal Reasoning Deficiency, Contextual Inconsistency, Unanswered, Output Error, and Deficiency in Medical Language Generation. Evaluation of 10 leading models reveals vulnerabilities: despite achieving 0.86 accuracy in medical knowledge recall, critical reasoning tasks show 96.3\% omission, while safety ethics evaluations expose alarming inconsistency (robustness score: 0.79) under option shuffled. Our analysis uncovers systemic weaknesses in knowledge boundary enforcement and multi-step reasoning. To address these, we propose a tiered optimization strategy spanning four levels—from prompt engineering and knowledge-augmented retrieval to hybrid neuro-symbolic architectures and causal reasoning frameworks. This work establishes an actionable roadmap for developing clinically robust LLMs while redefining evaluation paradigms through error-driven insights, ultimately advancing the safety and trustworthiness of AI in high-stakes medical environments.

UNSTRUCTURED The evaluation and improvement of medical large language models (LLMs) are critical for their real-world deployment, particularly in ensuring accuracy, safety, and ethical alignment. Existing frameworks inadequately dissect domain-specific error patterns or address cross-modal challenges. This study introduces a granular error taxonomy through systematic analysis of top 10 models on MedBench, categorizing incorrect responses into eight types: Omissions, Hallucination, Format Mismatch, Causal Reasoning Deficiency, Contextual Inconsistency, Unanswered, Output Error, and Deficiency in Medical Language Generation. Evaluation of 10 leading models reveals vulnerabilities: despite achieving 0.86 accuracy in medical knowledge recall, critical reasoning tasks show 96.3\% omission, while safety ethics evaluations expose alarming inconsistency (robustness score: 0.79) under option shuffled. Our analysis uncovers systemic weaknesses in knowledge boundary enforcement and multi-step reasoning. To address these, we propose a tiered optimization strategy spanning four levels—from prompt engineering and knowledge-augmented retrieval to hybrid neuro-symbolic architectures and causal reasoning frameworks. This work establishes an actionable roadmap for developing clinically robust LLMs while redefining evaluation paradigms through error-driven insights, ultimately advancing the safety and trustworthiness of AI in high-stakes medical environments.