This study was designed to assess how different prompt engineering techniques, specifically direct prompts, Chain of Thought (CoT), and a modified CoT approach, influence the ability of GPT-3.5 to answer clinical and calculation-based medical questions, particularly those styled like the USMLE Step 1 exams. To achieve this, we analyzed the responses of GPT-3.5 to two distinct sets of questions: a batch of 1000 questions generated by GPT-4, and another set comprising 95 real USMLE Step 1 questions. These questions spanned a range of medical calculations and clinical scenarios across various fields and difficulty levels. Our analysis revealed that there were no significant differences in the accuracy of GPT-3.5's responses when using direct prompts, CoT, or modified CoT methods. For instance, in the USMLE sample, the success rates were 61.7% for direct prompts, 62.8% for CoT, and 57.4% for modified CoT, with a p-value of 0.734. Similar trends were observed in the responses to GPT-4 generated questions, both clinical and calculation-based, with p-values above 0.05 indicating no significant difference between the prompt types. The conclusion drawn from this study is that the use of CoT prompt engineering does not significantly alter GPT-3.5's effectiveness in handling medical calculations or clinical scenario questions styled like those in USMLE exams. This finding is crucial as it suggests that performance of ChatGPT remains consistent regardless of whether a CoT technique is used instead of direct prompts. This consistency could be instrumental in simplifying the integration of AI tools like ChatGPT into medical education, enabling healthcare professionals to utilize these tools with ease, without the necessity for complex prompt engineering.

Large Language Models (LLMs) are becoming integral to healthcare analytics. However, the influence of the temperature hyperparameter, which controls output randomness, remains poorly understood in clinical tasks. This study evaluates the effects of different temperature settings across various clinical tasks. We conducted a retrospective cohort study using electronic health records from the Mount Sinai Health System, collecting a random sample of 1283 patients from January to December 2023. Three LLMs (GPT-4, GPT-3.5, and Llama-3-70b) were tested at five temperature settings (0.2, 0.4, 0.6, 0.8, 1.0) for their ability to predict in-hospital mortality (binary classification), length of stay (regression), and the accuracy of medical coding (clinical reasoning). For mortality prediction, all models’ accuracies were generally stable across different temperatures. Llama-3 showed the highest accuracy, around 90%, followed by GPT-4 (80-83%) and GPT-3.5 (74-76%). Regression analysis for predicting the length of stay showed that all models performed consistently across different temperatures. In the medical coding task, performance was also stable across temperatures, with GPT-4 achieving the highest accuracy at 17% for complete code accuracy. Our study demonstrates that LLMs maintain consistent accuracy across different temperature settings for varied clinical tasks, challenging the assumption that lower temperatures are necessary for clinical reasoning.

Background and aim: We analyzed an inclusive gradient boosting model to predict hospital admission from the emergency department (ED) at different time points. We compared its results to multiple models built exclusively at each time point. Methods: This retrospective multisite study utilized ED data from the Mount Sinai Health System, NY, during 2015–2019. Data included tabular clinical features and free-text triage notes represented using bag-of-words. A full gradient boosting model, trained on data available at different time points (30, 60, 90, 120, and 150 min), was compared to single models trained exclusively at data available at each time point. This was conducted by concatenating the rows of data available at each time point to one data matrix for the full model, where each row is considered a separate case. Results: The cohort included 1,043,345 ED visits. The full model showed comparable results to the single models at all time points (AUCs 0.84–0.88 for different time points for both the full and single models). Conclusion: A full model trained on data concatenated from different time points showed similar results to single models trained at each time point. An ML-based prediction model can use used for identifying hospital admission.

Background: Predicting hospitalization from nurse triage notes has the potential to augment care. However, there needs to be careful considerations for which models to choose for this goal. Specifically, health systems will have varying degrees of computational infrastructure available and budget constraints. Objective: To this end, we compared the performance of the deep learning, Bidirectional Encoder Representations from Transformers (BERT)–based model, Bio-Clinical-BERT, with a bag-of-words (BOW) logistic regression (LR) model incorporating term frequency–inverse document frequency (TF-IDF). These choices represent different levels of computational requirements. Methods: A retrospective analysis was conducted using data from 1,391,988 patients who visited emergency departments in the Mount Sinai Health System spanning from 2017 to 2022. The models were trained on 4 hospitals’ data and externally validated on a fifth hospital’s data. Results: The Bio-Clinical-BERT model achieved higher areas under the receiver operating characteristic curve (0.82, 0.84, and 0.85) compared to the BOW-LR-TF-IDF model (0.81, 0.83, and 0.84) across training sets of 10,000; 100,000; and ~1,000,000 patients, respectively. Notably, both models proved effective at using triage notes for prediction, despite the modest performance gap. Conclusions: Our findings suggest that simpler machine learning models such as BOW-LR-TF-IDF could serve adequately in resource-limited settings. Given the potential implications for patient care and hospital resource management, further exploration of alternative models and techniques is warranted to enhance predictive performance in this critical domain. International Registered Report Identifier (IRRID): RR2-10.1101/2023.08.07.23293699

Hematopoietic stem cell transplantation (HSCT) is becoming an increasingly recognized indication for treatment of autoimmune diseases and severe immune-mediated disorders. However, multicenter registry data have demonstrated higher than anticipated early toxicity, approximately 10% for autoimmune diseases in general, and 20–27% for diffuse systemic sclerosis (scleroderma). If uncorrected, this high treatment-related mortality will hinder development of stem cell therapy for immune-mediated diseases. In order to develop safer regimens, we address some pitfalls and concepts involved in design and selection of conditioning regimens for autoimmune diseases in general, and because it is associated with the highest regimen-related toxicity, scleroderma in specific.

Background: Hospital-acquired pressure injuries (HAPIs) affect approximately 2.5 million patients annually in the United States, leading to increased morbidity and healthcare costs. Current rule-based screening tools, such as the Braden Scale, lack sensitivity, highlighting the need for improved risk prediction methods. Methods: We developed and externally validated a machine learning model to predict HAPI risk using longitudinal electronic medical record (EMR) data. This study included adult inpatients (2018–2023) across five hospitals within a large health system. An automated pipeline was built for EMR data curation, labeling, and integration. The model employed XGBoost with recursive feature elimination to identify 35 optimal clinical variables and utilized time-series analysis for dynamic risk prediction. Results: Internal validation and multi-center external validation on 5510 hospitalizations demonstrated AUROC values of 0.83–0.85. The model outperformed the Braden Scale in sensitivity and F1-score and showed superior performance compared to previous predictive models. Conclusions: This is the first externally validated, cross-institutional HAPI prediction model using longitudinal EMR data and automated pipelines. The model demonstrates strong generalizability, scalability, and real-time applicability, offering a novel bioengineering approach to improve HAPI prevention, patient care, and clinical operations.