Background: The advancement of large language models (LLMs) offers significant opportunities for health care, particularly in the generation of medical documentation. However, challenges related to ensuring the accuracy and reliability of LLM outputs, coupled with the absence of established quality standards, have raised concerns about their clinical application. Objective: This study aimed to develop and validate an evaluation framework for assessing the accuracy and clinical applicability of LLM-generated emergency department (ED) records, aiming to enhance artificial intelligence integration in health care documentation. Methods: We organized the Healthcare Prompt-a-thon, a competitive event designed to explore the capabilities of LLMs in generating accurate medical records. The event involved 52 participants who generated 33 initial ED records using HyperCLOVA X, a Korean-specialized LLM. We applied a dual evaluation approach. First, clinical evaluation: 4 medical professionals evaluated the records using a 5-point Likert scale across 5 criteria—appropriateness, accuracy, structure/format, conciseness, and clinical validity. Second, quantitative evaluation: We developed a framework to categorize and count errors in the LLM outputs, identifying 7 key error types. Statistical methods, including Pearson correlation and intraclass correlation coefficients (ICC), were used to assess consistency and agreement among evaluators. Results: The clinical evaluation demonstrated strong interrater reliability, with ICC values ranging from 0.653 to 0.887 (P<.001), and a test-retest reliability Pearson correlation coefficient of 0.776 (P<.001). Quantitative analysis revealed that invalid generation errors were the most common, constituting 35.38% of total errors, while structural malformation errors had the most significant negative impact on the clinical evaluation score (Pearson r=–0.654; P<.001). A strong negative correlation was found between the number of quantitative errors and clinical evaluation scores (Pearson r=–0.633; P<.001), indicating that higher error rates corresponded to lower clinical acceptability. Conclusions: Our research provides robust support for the reliability and clinical acceptability of the proposed evaluation framework. It underscores the framework’s potential to mitigate clinical burdens and foster the responsible integration of artificial intelligence technologies in health care, suggesting a promising direction for future research and practical applications in the field.

Background: Bisphosphonate-induced osteonecrosis of the jaw (BRONJ) presents with a typical pattern of jaw necrosis in patients who have been prescribed bisphosphonates (BPs) and other antiangiogenetic drugs to treat osteoporosis or bone-related complications of cancer. Methods: This study divided 38 patients with BRONJ into two groups according to the prescribing causes: cancer (n = 13) and osteoporosis (n = 25), and underwent whole exome sequencing and compared them with normal controls (n = 90). To identify candidate genes and variants, we conducted three analyses: a traditional genetic model, gene-wise variant score burden, and rare-variant analysis methods. Results: The stop-gain mutation (rs117889746) of the PZP gene in the BRONJ cancer group was significantly identified in the additive trend model analysis. In the cancer group, ARIDS, HEBP1, LTBP1, and PLVAP were identified as candidate genes. In the osteoporosis group, VEGFA, DFFA, and FAM193A genes showed a significant association. No significant genes were identified in the rare-variant analysis pipeline. Biologically accountable functions related to BRONJ occurrence-angiogenesis-related signaling (VEGFA and PLVAP genes), TGF-β signaling (LTBP1 and PZP genes), heme toxicity (HEBP1) and osteoblast maturation (ARIDS)-were shown in candidate genes. Conclusion: This study showed that the candidate causative genes contributing to the development of BRONJ differ according to the BP dose and background disease.

Objective. We propose 2 Medical Dictionary for Regulatory Activities–enabled pharmacovigilance algorithms, MetaLAB and MetaNurse, powered by a per-year meta-analysis technique and improved subject sampling strategy. Matrials and methods. This study developed 2 novel algorithms, MetaLAB for laboratory abnormalities and MetaNurse for standard nursing statements, as significantly improved versions of our previous electronic health record (EHR)–based pharmacovigilance method, called CLEAR. Adverse drug reaction (ADR) signals from 117 laboratory abnormalities and 1357 standard nursing statements for all precautionary drugs (n = 101) were comprehensively detected and validated against SIDER (Side Effect Resource) by MetaLAB and MetaNurse against 11 817 and 76 457 drug-ADR pairs, respectively. Results. We demonstrate that MetaLAB (area under the curve, AUC = 0.61 ± 0.18) outperformed CLEAR (AUC = 0.55 ± 0.06) when we applied the same 470 drug-event pairs as the gold standard, as in our previous research. Receiver operating characteristic curves for 101 precautionary terms in the Medical Dictionary for Regulatory Activities Preferred Terms were obtained for MetaLAB and MetaNurse (0.69 ± 0.11; 0.62 ± 0.07), which complemented each other in terms of ADR signal coverage. Novel ADR signals discovered by MetaLAB and MetaNurse were successfully validated against spontaneous reports in the US Food and Drug Administration Adverse Event Reporting System database. Discussion. The present study demonstrates the symbiosis of laboratory test results and nursing statements for ADR signal detection in terms of their system organ class coverage and performance profiles. Conclusion. Systematic discovery and evaluation of the wide spectrum of ADR signals using standard-based observational electronic health record data across many institutions will affect drug development and use, as well as postmarketing surveillance and regulation.