Ongoing research attempts to benchmark large language models (LLM) against physicians’ fund of knowledge by assessing LLM performance on medical examinations. No prior study has assessed LLM performance on internal medicine (IM) board examination questions. Limited data exists on how knowledge supplied to the models, derived from medical texts improves LLM performance. The performance of GPT-3.5, GPT-4.0, LaMDA and Llama 2, with and without additional model input augmentation, was assessed on 240 randomly selected IM board-style questions. Questions were sourced from the Medical Knowledge Self-Assessment Program released by the American College of Physicians with each question serving as part of the LLM prompt. When available, LLMs were accessed both through their application programming interface (API) and their corresponding chatbot. Mode inputs were augmented with Harrison’s Principles of Internal Medicine using the method of Retrieval Augmented Generation. LLM-generated explanations to 25 correctly answered questions were presented in a blinded fashion alongside the MKSAP explanation to an IM board-certified physician tasked with selecting the human generated response. GPT-4.0, accessed either through Bing Chat or its API, scored 77.5–80.7% outperforming GPT-3.5, human respondents, LaMDA and Llama 2 in that order. GPT-4.0 outperformed human MKSAP users on every tested IM subject with its highest and lowest percentile scores in Infectious Disease (80th) and Rheumatology (99.7th), respectively. There is a 3.2–5.3% decrease in performance of both GPT-3.5 and GPT-4.0 when accessing the LLM through its API instead of its online chatbot. There is 4.5–7.5% increase in performance of both GPT-3.5 and GPT-4.0 accessed through their APIs after additional input augmentation. The blinded reviewer correctly identified the human generated MKSAP response in 72% of the 25-question sample set. GPT-4.0 performed best on IM board-style questions outperforming human respondents. Augmenting with domain-specific information improved performance rendering Retrieval Augmented Generation a possible technique for improving accuracy in medical examination LLM responses.

The verticillium fungus has become a widespread threat to olive fields around the world in recent years. The accurate and early detection of the disease at scale could support solving the problem. In this paper, we use the YOLO version 5 model to detect verticillium fungus in olive trees using aerial RGB imagery captured by unmanned aerial vehicles. The aim of our paper is to compare different architectures of the model and evaluate their performance on this task. The architectures are evaluated at two different input sizes each through the most widely used metrics for object detection and classification tasks (precision, recall, mAP@0.5 and mAP@0.5:0.95). Our results show that the YOLOv5 algorithm is able to deliver good results in detecting olive trees and predicting their status, with the different architectures having different strengths and weaknesses.

Introduction: Contemporary efforts to predict surgical outcomes focus on the associations between traditional discrete surgical risk factors. We aimed to determine whether natural language processing (NLP) of unstructured operative notes improves the prediction of residual disease in women with advanced epithelial ovarian cancer (EOC) following cytoreductive surgery. Methods: Electronic Health Records were queried to identify women with advanced EOC including their operative notes. The Term Frequency – Inverse Document Frequency (TF-IDF) score was used to quantify the discrimination capacity of sequences of words (n-grams) regarding the existence of residual disease. We employed the state-of-the-art RoBERTa-based classifier to process unstructured surgical notes. Discrimination was measured using standard performance metrics. An XGBoost model was then trained on the same dataset using both discrete and engineered clinical features along with the probabilities outputted by the RoBERTa classifier. Results: The cohort consisted of 555 cases of EOC cytoreduction performed by eight surgeons between January 2014 and December 2019. Discrete word clouds weighted by n-gram TF-IDF score difference between R0 and non-R0 resection were identified. The words ‘adherent’ and ‘miliary disease’ best discriminated between the two groups. The RoBERTa model reached high evaluation metrics (AUROC .86; AUPRC .87, precision, recall, and F1 score of .77 and accuracy of .81). Equally, it outperformed models that used discrete clinical and engineered features and outplayed the performance of other state-of-the-art NLP tools. When the probabilities from the RoBERTa classifier were combined with commonly used predictors in the XGBoost model, a marginal improvement in the overall model’s performance was observed (AUROC and AUPRC of .91, with all other metrics the same). Conclusion/Implications: We applied a sui generis approach to extract information from the abundant textual surgical data and demonstrated how it can be effectively used for classification prediction, outperforming models relying on conventional structured data. State-of-art NLP applications in biomedical texts can improve modern EOC care.

Large language models possess tremendous natural language understanding and generation abilities. However, they often lack the ability to discern between fact and fiction, leading to factually incorrect responses. Open Government Data are repositories of, often times linked, information that is freely available to everyone. By combining these two technologies in a proof of concept designed application utilizing the GPT3.5 OpenAI model and the Scottish open statistics portal, we show that not only is it possible to augment the large language model’s factuality of responses, but also propose a novel way to effectively access and retrieve statistical information from the data portal just through natural language querying. We anticipate that this paper will trigger a discussion regarding the transformation of Open Government Portals through large language models.

The Surgical Complexity Score (SCS) has been widely used to describe the surgical effort during advanced stage epithelial ovarian cancer (EOC) cytoreduction. Referring to a variety of multi-visceral resections, it best combines the numbers with the complexity of the sub-procedures. Nevertheless, not all potential surgical procedures are described by this score. Lately, the European Society for Gynaecological Oncology (ESGO) has established standard outcome quality indicators pertinent to achieving complete cytoreduction (CC0). There is a need to define what weight all these surgical sub-procedures comprising CC0 would be given. Prospectively collected data from 560 surgically cytoreduced advanced stage EOC patients were analysed at a UK tertiary referral centre.We adapted the structured ESGO ovarian cancer report template. We employed the eXtreme Gradient Boosting (XGBoost) algorithm to model a long list of surgical sub-procedures. We applied the Shapley Additive explanations (SHAP) framework to provide global (cohort) explainability. We used Cox regression for survival analysis and constructed Kaplan-Meier curves. The XGBoost model predicted CC0 with an acceptable accuracy (area under curve [AUC] = 0.70; 95% confidence interval [CI] = 0.63–0.76). Visual quantification of the feature importance for the prediction of CC0 identified upper abdominal peritonectomy (UAP) as the most important feature, followed by regional lymphadenectomies. The UAP best correlated with bladder peritonectomy and diaphragmatic stripping (Pearson’s correlations > 0.5). Clear inflection points were shown by pelvic and para-aortic lymph node dissection and ileocecal resection/right hemicolectomy, which increased the probability for CC0. When UAP was solely added to a composite model comprising of engineered features, it substantially enhanced its predictive value (AUC = 0.80, CI = 0.75–0.84). The UAP was predictive of poorer progression-free survival (HR = 1.76, CI 1.14–2.70, P: 0.01) but not overall survival (HR = 1.06, CI 0.56–1.99, P: 0.86). The SCS did not have significant survival impact. Machine Learning allows for operational feature selection by weighting the relative importance of those surgical sub-procedures that appear to be more predictive of CC0. Our study identifies UAP as the most important procedural predictor of CC0 in surgically cytoreduced advanced-stage EOC women. The classification model presented here can potentially be trained with a larger number of samples to generate a robust digital surgical reference in high output tertiary centres. The upper abdominal quadrants should be thoroughly inspected to ensure that CC0 is achievable.