Jan Griewatz

Jan Griewatz

Auto-generated profile, based on publicly available information

Jan Griewatz

A Competence Centre for University Teaching in Medicine Baden-Wuerttemberg, Faculty of Medicine , University of Tuebingen , Tuebingen , Germany

Research Focus & Metrics

Academic Career Length

12Years (2015-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
16 January 2024 in JMIR Medical Education
Verified
Journal Article
Open access
FHFriederike Holderried
Friederike Holderried
CSChristian Stegemann–Philipps
Christian Stegemann–Philipps
LHLea Herschbach
Lea Herschbach
JMJulia-Astrid Moldt
Julia-Astrid Moldt
ANAndrew Nevins
Andrew Nevins
MMMoritz Mahling
Moritz Mahling
Background: Communication is a core competency of medical professionals and of utmost importance for patient safety. Although medical curricula emphasize communication training, traditional formats, such as real or simulated patient interactions, can present psychological stress and are limited in repetition. The recent emergence of large language models (LLMs), such as generative pretrained transformer (GPT), offers an opportunity to overcome these restrictions Objective: The aim of this study was to explore the feasibility of a GPT-driven chatbot to practice history taking, one of the core competencies of communication. Methods: We developed an interactive chatbot interface using GPT-3.5 and a specific prompt including a chatbot-optimized illness script and a behavioral component. Following a mixed methods approach, we invited medical students to voluntarily practice history taking. To determine whether GPT provides suitable answers as a simulated patient, the conversations were recorded and analyzed using quantitative and qualitative approaches. We analyzed the extent to which the questions and answers aligned with the provided script, as well as the medical plausibility of the answers. Finally, the students filled out the Chatbot Usability Questionnaire (CUQ). Results: A total of 28 students practiced with our chatbot (mean age 23.4, SD 2.9 years). We recorded a total of 826 question-answer pairs (QAPs), with a median of 27.5 QAPs per conversation and 94.7% (n=782) pertaining to history taking. When questions were explicitly covered by the script (n=502, 60.3%), the GPT-provided answers were mostly based on explicit script information (n=471, 94.4%). For questions not covered by the script (n=195, 23.4%), the GPT answers used 56.4% (n=110) fictitious information. Regarding plausibility, 842 (97.9%) of 860 QAPs were rated as plausible. Of the 14 (2.1%) implausible answers, GPT provided answers rated as socially desirable, leaving role identity, ignoring script information, illogical reasoning, and calculation error. Despite these results, the CUQ revealed an overall positive user experience (77/100 points). Conclusions: Our data showed that LLMs, such as GPT, can provide a simulated patient experience and yield a good user experience and a majority of plausible answers. Our analysis revealed that GPT-provided answers use either explicit script information or are based on available information, which can be understood as abductive reasoning. Although rare, the GPT-based chatbot provides implausible information in some instances, with the major tendency being socially desirable instead of medically plausible information.

1 April 2017 in Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen
Verified
Journal Article
Closed access
AHAnne Herrmann-Werner
Anne Herrmann-Werner
RGRegina Gramer
Regina Gramer
RERebecca Erschens
Rebecca Erschens
CNChristoph Nikendei
Christoph Nikendei
AWAnnette Wosnik
Annette Wosnik
FJFlorian Junne
Florian Junne

23 January 2024 in Journal of Medical Internet Research
Verified
Journal Article
Open access
AHAnne Herrmann-Werner
Anne Herrmann-Werner
TLTeresa Loda
Teresa Loda
FHFriederike Holderried
Friederike Holderried
LHLea Herschbach
Lea Herschbach
JGJan Griewatz
Jan Griewatz
MMMoritz Mahling
Moritz Mahling
Background: Large language models such as GPT-4 (Generative Pre-trained Transformer 4) are being increasingly used in medicine and medical education. However, these models are prone to “hallucinations” (ie, outputs that seem convincing while being factually incorrect). It is currently unknown how these errors by large language models relate to the different cognitive levels defined in Bloom’s taxonomy. Objective: This study aims to explore how GPT-4 performs in terms of Bloom’s taxonomy using psychosomatic medicine exam questions. Methods: We used a large data set of psychosomatic medicine multiple-choice questions (N=307) with real-world results derived from medical school exams. GPT-4 answered the multiple-choice questions using 2 distinct prompt versions: detailed and short. The answers were analyzed using a quantitative approach and a qualitative approach. Focusing on incorrectly answered questions, we categorized reasoning errors according to the hierarchical framework of Bloom’s taxonomy. Results: GPT-4’s performance in answering exam questions yielded a high success rate: 93% (284/307) for the detailed prompt and 91% (278/307) for the short prompt. Questions answered correctly by GPT-4 had a statistically significant higher difficulty than questions answered incorrectly (P=.002 for the detailed prompt and P<.001 for the short prompt). Independent of the prompt, GPT-4’s lowest exam performance was 78.9% (15/19), thereby always surpassing the “pass” threshold. Our qualitative analysis of incorrect answers, based on Bloom’s taxonomy, showed that errors were primarily in the “remember” (29/68) and “understand” (23/68) cognitive levels; specific issues arose in recalling details, understanding conceptual relationships, and adhering to standardized guidelines. Conclusions: GPT-4 demonstrated a remarkable success rate when confronted with psychosomatic medicine multiple-choice exam questions, aligning with previous findings. When evaluated through Bloom’s taxonomy, our data revealed that GPT-4 occasionally ignored specific facts (remember), provided illogical reasoning (understand), or failed to apply concepts to a new situation (apply). These errors, which were confidently presented, could be attributed to inherent model biases and the tendency to generate outputs that maximize likelihood.

28 May 2019 in BMC Medical Education
Verified
Journal Article
Open access
PDF file
NCNora Celebi
Nora Celebi
JGJan Griewatz
Jan Griewatz
NMNisar P. Malek
Nisar P. Malek
SKSarah Krieg
Sarah Krieg
TKToni Kuehnl
Toni Kuehnl
EFEckhart Fröhlich
Eckhart Fröhlich
Ultrasound is one of the most important imaging techniques in clinical medicine with unique advantages. Skills in ultrasound imaging are very usefull for physicians including novices and thus also mandated by the Task Force “National Competence-Based Learning Objectives for Undergraduate Medical Education” (NKLM) in Germany and as well as by the German Ultrasound Society (Deutsche Gesellschaft für Ultraschall in der Medizin, DEGUM). Since ultrasound is best learned hands-on in very small supervised groups, we developed and implemented a comprehensive ultrasound-curriculum for all undergraduate medical students of our faculty using a peer-teaching concept. We used Kern‘s six-step model of curricular development comprising (1) problem identification and general needs assessment, (2) needs assessment of the targeted learners, (3) goals and objectives, (4) educational stategies, (5) implementation, and (6) evaluation and feedback. The developed curriculum covers basic ultrasound of the abdomen and the throat, eFAST (Extended Focused Assessment with Sonography for Trauma), lung-ultrasound, FEEL (Focused Echocardiography in Emergency Life Support) and compression duplex sonography of the thigh deep vein system. All 5th year medical students receive a 90 min lecture on ultrasound basics by a faculty member and then a 12.5 h hands-on course divided into three sessions with one student tutor for every 4 students. The students are provided with a script (PDF-File) that covers all the learning goals, including example images of pathologies. The student tutors are trained during a 1 week ultrasound course and a 21-day rotation through seven different ultrasound laboratories. In addition, they undergo a standardized 1.5 day didactical training. Prior to the implementation for all students, the overall course was tested on 27 volunteer students. These students rated (on a 6-point Likert scale from 1 = excellent to 6 = very poor) the satisfaction with the student tutors and the faculty members as 1.4 ± .9 (mean ± stddev) and 1.3 ± .5 respectively. A comprehensive ultrasound curriculum for all undergraduate medical students using a peer-teaching concept is feasible. Further studies are needed to evaluate in detail the learning outcomes for students and student tutors.

25 November 2015 in Journal of Medical Internet Research
Verified
Journal Article
Open access
MBMartina Bientzle
Martina Bientzle
JGJan Griewatz
Jan Griewatz
Joachim KimmerleJoachim Kimmerle
Joachim Kimmerle
JKJulia Küppers
Julia Küppers
UCUlrike Cress
Ulrike Cress
MLMaria Lammerding-Koeppel
Maria Lammerding-Koeppel
Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet

16 November 2017 in Medical Teacher
Verified
Journal Article
Closed access
MLMaria Lammerding-Koeppel
Maria Lammerding-Koeppel
OFOlaf Fritze
Olaf Fritze
MGMarianne Giesler
Marianne Giesler
ENElisabeth Narciss
Elisabeth Narciss
SSSandra Steffens
Sandra Steffens
JGJan Griewatz
Jan Griewatz
Objectives: Internationally, scientific and research-related competencies need to be sufficiently targeted as core outcomes in many undergraduate medical curricula. Since 2015, standards have been recommended for Germany in the National Competency-based Learning Objective Catalogue in Medicine (NKLM). The aim of this study is to develop a multi-center mapping approach for curricular benchmarking against national standards and against other medical faculties. Method: A total of 277 faculty members from four German medical faculties have mapped the local curriculum against the scientific and research-related NKLM objectives, using consented procedures, metrics, and tools. The amount of mapping citations of each objective is used as indicator for its weighting in the local curriculum. Achieved competency levels after five-year education are compared. Results: All four programs fulfill the NKLM standards, with each emphasizing different sub-competencies explicitly in writing (Scholar: 17–41% of all courses; Medical Scientific Skills: 14–37% of all courses). Faculties show major or full agreement in objective weighting: Scholar 44%, scientific skills 79%. The given NKLM competency level is met or even outperformed in 78–100% of the courses. Conclusions: The multi-center mapping approach provides an informative dataset allowing curricular diagnosis by external benchmarking and guidance for optimization of local curricula.