Integrating Large Language Models (LLMs) into healthcare promises substantial advancements but requires careful consideration of technical, ethical, and regulatory challenges. Closed LLMs of private companies offer ease of deployment but pose risks related to data privacy and vendor dependence. Open LLMs deployed on local hardware enable greater model customization but demand resources and technical expertise. Balancing these approaches, with collaboration among clinicians, researchers, and companies is crucial to ensure effective, secure, and ethical implementation.

Background: Systematically screening published literature to determine the relevant publications to synthesize in a review is a time-consuming and difficult task. Large language models (LLMs) are an emerging technology with promising capabilities for the automation of language-related tasks that may be useful for such a purpose. Methods: LLMs were used as part of an automated system to evaluate the relevance of publications to a certain topic based on defined criteria and based on the title and abstract of each publication. A Python script was created to generate structured prompts consisting of text strings for instruction, title, abstract, and relevant criteria to be provided to an LLM. The relevance of a publication was evaluated by the LLM on a Likert scale (low relevance to high relevance). By specifying a threshold, different classifiers for inclusion/exclusion of publications could then be defined. The approach was used with four different openly available LLMs on ten published data sets of biomedical literature reviews and on a newly human-created data set for a hypothetical new systematic literature review. Results: The performance of the classifiers varied depending on the LLM being used and on the data set analyzed. Regarding sensitivity/specificity, the classifiers yielded 94.48%/31.78% for the FlanT5 model, 97.58%/19.12% for the OpenHermes-NeuralChat model, 81.93%/75.19% for the Mixtral model and 97.58%/38.34% for the Platypus 2 model on the ten published data sets. The same classifiers yielded 100% sensitivity at a specificity of 12.58%, 4.54%, 62.47%, and 24.74% on the newly created data set. Changing the standard settings of the approach (minor adaption of instruction prompt and/or changing the range of the Likert scale from 1–5 to 1–10) had a considerable impact on the performance. Conclusions: LLMs can be used to evaluate the relevance of scientific publications to a certain review topic and classifiers based on such an approach show some promising results. To date, little is known about how well such systems would perform if used prospectively when conducting systematic literature reviews and what further implications this might have. However, it is likely that in the future researchers will increasingly use LLMs for evaluating and classifying scientific publications.

Technological progress of machine learning and natural language processing has led to the development of large language models (LLMs), capable of producing well-formed text responses and providing natural language access to knowledge. Modern conversational LLMs such as ChatGPT have shown remarkable capabilities across a variety of fields, including medicine. These models may assess even highly specialized medical knowledge within specific disciplines, such as radiation therapy. We conducted an exploratory study to examine the capabilities of ChatGPT to answer questions in radiation therapy. A set of multiple-choice questions about clinical, physics, and biology general knowledge in radiation oncology as well as a set of open-ended questions were created. These were given as prompts to the LLM ChatGPT, and the answers were collected and analyzed. For the multiple-choice questions, it was checked how many of the answers of the model could be clearly assigned to one of the allowed multiple-choice-answers, and the proportion of correct answers was determined. For the open-ended questions, independent blinded radiation oncologists evaluated the quality of the answers regarding correctness and usefulness on a 5-point Likert scale. Furthermore, the evaluators were asked to provide suggestions for improving the quality of the answers. For 70 multiple-choice questions, ChatGPT gave valid answers in 66 cases (94.3%). In 60.61% of the valid answers, the selected answer was correct (50.0% of clinical questions, 78.6% of physics questions, and 58.3% of biology questions). For 25 open-ended questions, 12 answers of ChatGPT were considered as “acceptable,” “good,” or “very good” regarding both correctness and helpfulness by all 6 participating radiation oncologists. Overall, the answers were considered “very good” in 29.3% and 28%, “good” in 28% and 29.3%, “acceptable” in 19.3% and 19.3%, “bad” in 9.3% and 9.3%, and “very bad” in 14% and 14% regarding correctness/helpfulness. Modern conversational LLMs such as ChatGPT can provide satisfying answers to many relevant questions in radiation therapy. As they still fall short of consistently providing correct information, it is problematic to use them for obtaining medical information. As LLMs will further improve in the future, they are expected to have an increasing impact not only on general society, but also on clinical practice, including radiation oncology.

Background: Artificial intelligence continues to transform health care, offering promising applications in clinical practice and medical education. While large language models (LLMs), as a form of generative artificial intelligence, have shown potential to match or surpass medical students in licensing examinations, their performance varies across languages. Recent studies highlight the complex influence and interdependency of factors such as language and model type on LLMs’ accuracy; yet, cross-language comparisons remain underexplored. Objective: This study evaluates the performance of LLMs in answering medical multiple-choice questions quantitatively and qualitatively across 3 languages (German, French, and Italian), aiming to uncover model capabilities in a multilingual medical education context. Methods: For this mixed methods study, 114 publicly accessible multiple-choice questions in German, French, and Italian from an online self-assessment tool were analyzed. A quantitative performance analysis of several LLMs developed by OpenAI, Meta AI, Anthropic, and DeepSeek was conducted to evaluate their performance on answering the questions in text-only format. For the comparative analysis, a variation of input question language (German, French, and Italian) and prompt language (English vs language-matched) was used. The 2 best-performing LLMs were then prompted to provide answer explanations for incorrectly answered questions. A subsequent qualitative analysis was conducted on these explanations to identify the reasons leading to the incorrect answers. Results: The performance of LLMs in answering medical multiple-choice questions varied by model and language, showing substantial differences in accuracy (between 64% and 87%). The effect of input question language was significant ( P <.01) with models performing best on German questions. Across the analyzed LLMs, prompting in English generally led to better performance in comparison to language-matched prompts, but the top-performing models exceptionally showed comparable results for language-matched prompts. Qualitative analysis revealed that answer explanations of the analyzed models (GPT4o and Claude-Sonnet-3.7) showed different reasoning errors. In several explanations, this occurred despite factual accuracy on the represented topic. Furthermore, this analysis revealed 3 questions to be insufficiently precise. Conclusions: Our results underline the potential of LLMs in answering medical examination questions and highlight the importance of careful consideration of model choice, prompt, and input languages, because of relevant performance variability across these factors. Analysis of answer explanations demonstrates a valuable use case of LLMs for improving examination question quality in medical education, if data security regulations permit their use. Human oversight of language-sensitive or clinically nuanced content remains essential to determine whether incorrect output stems from flaws in the questions themselves or from errors generated by the LLMs. There is a need for ongoing evaluation as well as transparent reporting to ensure reliable integration of LLMs into medical education contexts.

In oncology, decision-making in individual situations is often very complex. To deal with such complexity, people tend to reduce it by relying on their initial intuition. The downside of this intuitive, subjective way of decision-making is that it is prone to cognitive and emotional biases such as overestimating the quality of its judgements or being influenced by one’s current mood. Hence, clinical predictions based on intuition often turn out to be wrong and to be outperformed by statistical predictions. Structuring and objectivizing oncological decision-making may thus overcome some of these issues and have advantages such as avoidance of unwarranted clinical practice variance or error-prevention. Even for uncertain situations with limited medical evidence available or controversies about the best treatment option, structured decision-making approaches like clinical algorithms could outperform intuitive decision-making. However, the idea of such algorithms is not to prescribe the clinician which decision to make nor to abolish medical judgement, but to support physicians in making decisions in a systematic and structured manner. An example for a use-case scenario where such an approach may be feasible is the selection of treatment dose in radiation oncology. In this paper, we will describe how a clinical algorithm for selection of a fractionation scheme for palliative irradiation of bone metastases can be created. We explain which steps in the creation process of a clinical algorithm for supporting decision-making need to be performed and which challenges and limitations have to be considered.

Chest radiography, computed tomography (CT) and positron emission tomography (PET)-CT are required for staging nonsmall cell lung cancers. Stage I cancers may be up to 4 cm in maximal diameter, with stage IA tumours being up to 3 cm and stage IB up to 4 cm. A lung cancer becomes stage II if the tumour is between 4 and ≤5 cm (stage IIA), or it spreads to ipsilateral peribronchial or hilar lymph nodes (stage IIB).Stage IA tumours should be surgically resected, ideally using minimally invasive methods. Lobectomy is usually performed, although some studies have shown good outcomes for sublobar resections. If surgery is not possible, stereotactic body radiotherapy is a good alternative. This involves delivering a few high-dose radiation treatments at very high precision. For stage IB to IIB disease, combinations of surgery, chemotherapy or immunotherapy and radiotherapy are used. There is evidence that neoadjuvant treatment (immunotherapy with nivolumab and chemotherapy for stage IB and II) optimises outcomes. Adjuvant chemotherapy with a platinum-based doublet (typically cisplatin+vinorelbine) should be offered for resected stage IIB tumours and considered for resected IIA tumours. Adjuvant pembrolizumab is used for stage IB–IIIA following resection and adjuvant platinum-based chemotherapy. Osimertinib may be used for resected stage IB to IIIA cancers which have relevant mutations (epidermal growth factor receptor exon 19 deletions or exon 21 (L858R) substitution). There are no fixed guidelines for follow-up, but most centres recommend 6-monthly CT scanning for the first 2–3 years after definitive treatment, followed by annual scans.