Manually creating multiple-choice questions (MCQ) is inefficient. Automatic item generation (AIG) offers a scalable solution, with two main approaches: template-based and non-template-based (AI-driven). Template-based AIG ensures accuracy but requires significant expert input to develop templates. In contrast, AI-driven AIG can generate questions quickly but with inaccuracies. The Hybrid AIG combines the strengths of both methods. However, neither have MCQs been generated using the Hybrid AIG approach nor has any validity evidence been provided. We generated MCQs using the Hybrid AIG approach and investigated the validity evidence of these questions by determining whether experts could identify the correct answers. We used a custom ChatGPT to develop an item template, which were then fed into Gazitor, a template-based AIG (non-AI) software. A panel of medical doctors identified the answers. Of 105 decisions, 101 (96.2%) matched the software's correct answer. In all MCQs (100%), the experts reached a consensus on the correct answer. The evidence corresponds to the 'Relations to Other Variables' in Messick's validity framework. The Hybrid AIG approach can enhance the efficiency of MCQ generation while maintaining accuracy. It mitigates concerns about hallucinations while benefiting from AI.

We aimed to determine the quality of AI-generated (ChatGPT-4 and Claude 3) Script Concordance Test (SCT) items through an expert panel. We generated SCT items on abdominal radiology using a complex prompt in large language model (LLM) chatbots (ChatGPT-4 and Claude 3 (Sonnet) in April 2024) and evaluated the items’ quality through an expert panel of 16 radiologists. Expert panel, which was blind to the origin of the items provided without modifications, independently answered each item and assessed them using 12 quality indicators. Data analysis included descriptive statistics, bar charts to compare responses against accepted forms, and a heatmap to show performance in terms of the quality indicators. SCT items generated by chatbots assess clinical reasoning rather than only factual recall (ChatGPT: 92.50%, Claude: 85.00%). The heatmap indicated that the items were generally acceptable, with most responses favorable across quality indicators (ChatGPT: 71.77%, Claude: 64.23%). The comparison of the bar charts with acceptable and unacceptable forms revealed that 73.33% and 53.33% of the questions in the items can be considered acceptable, respectively, for ChatGPT and Claude. The use of LLMs to generate SCT items can be helpful for medical educators by reducing the required time and effort. Although the prompt provides a good starting point, it remains crucial to review and revise AI-generated SCT items before educational use. The prompt and the custom GPT, “Script Concordance Test Generator”, available at https://chatgpt.com/g/g-RlzW5xdc1-script-concordance-test-generator, can streamline SCT item development. Nuestro objetivo fue determinar la calidad de los ítems de la Prueba de Concordancia de Guión (SCT) generada por IA (ChatGPT-4 y Claude 3) a través de un panel de expertos. Generamos ítems de SCT sobre radiología abdominal utilizando un mensaje complejo en chatbots de modelo de lenguaje grande (LLM) (ChatGPT-4 y Claude 3 (Sonnet) en abril de 2024) y evaluamos la calidad de los ítems a través de un panel de expertos de 16 radiólogos. El panel de expertos, que desconocía el origen de los ítems proporcionados sin modificaciones, respondió de forma independiente cada ítem y los evaluó utilizando 12 indicadores de calidad. El análisis de datos incluyó estadísticas descriptivas, gráficos de barras para comparar las respuestas con los formularios aceptados y un mapa de calor para mostrar el desempeño en términos de los indicadores de calidad. Los ítems de SCT generados por chatbots evalúan el razonamiento clínico en lugar de solo el recuerdo fáctico (ChatGPT: 92,50 %, Claude: 85,00 %). El mapa de calor indicó que los ítems eran generalmente aceptables, con la mayoría de las respuestas favorables en todos los indicadores de calidad (ChatGPT: 71,77 %, Claude: 64,23 %). La comparación de los gráficos de barras con formas aceptables e inaceptables reveló que el 73,33 % y el 53,33 % de las preguntas en los ítems pueden considerarse aceptables, respectivamente, para ChatGPT y Claude. El uso de LLM para generar ítems de SCT puede ser útil para los educadores médicos al reducir el tiempo y el esfuerzo necesarios. Aunque el mensaje proporciona un buen punto de partida, sigue siendo crucial revisar y modificar los ítems de SCT generados por IA antes de su uso educativo. El mensaje y el GPT personalizado, “Script Concordance Test Generator”, disponible en https://chatgpt.com/g/g-RlzW5xdc1-script-concordance-test-generator, pueden agilizar el desarrollo de ítems de SCT.

Medical education always evolves to incorporate more tools for specific needs in assessing clinical reasoning skills. Among these tools, Script Concordance Test (SCT) has a particular importance due to its focus on assessing decision-making in uncertain clinical situations. However, development of SCT items is effortful. Artificial intelligence tools, such as large language models, offer significant benefits. These models are already used for generating multiple-choice questions, and their use in generating SCTs offers great promise. However, this requires well-designed prompts to generate SCTs. This article proposes a generic prompt for the ChatGPT-4, Claude 3, Llama 3, and ChatGPT-4o large language model chatbots to generate SCTs, which can be tailored to various fields of medicine and different stages of medical education. It can help to streamline the development process of SCTs. Initial findings are promising, and there is a need for generating SCTs using large language models and conducting research to assess the quality of SCTs.

Background: Medical records, encompassing patient histories, progress notes, and more, play a crucial role in patient care and treatment, healthcare communication, medico-legal matters, and supporting financial documentation. Objective: Despite their significance, literature suggests inconsistencies in record quality and insufficient formal medical record-keeping education for medical students and residents. The study aimed to identify and evaluate the effectiveness of educational interventions by conducting a systematic review. Method: A literature search covering 2003–2023 and review following Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines was undertaken. Results: The literature search identified 44 relevant studies for inclusion. Educational methods, including lectures, feedback, workshops and discussions, addressed different components of the clinical record. The review revealed positive impacts on participant satisfaction, skills and attitudes related to record-keeping. However, some studies reported no significant positive outcomes, emphasising the need for higher-level evidence. Most studies adopted a single-group pretest–posttest design, presenting challenges in control group implementation. The Kirkpatrick evaluation levels were primarily at level 2, with few studies reaching level 3. The absence of studies at level 4 suggested the need for more robust evidence. Studies targeted medical residents more frequently than medical students, with a lack of interventions during the first year of medical education. Conclusion: Despite limitations including language bias and methodological variations, the review revealed diverse educational strategies and highlighted the necessity for more randomised controlled trials and studies providing higher-level evidence to enhance clinical record-keeping skills among medical students and residents. Implications: Medical record-keeping educational interventions can significantly improve the documentation skills of medical students and residents, thereby enhancing patient care, communication and medico-legal compliance.