Emi Furukawa

Dr. Emi Furukawa

The University of Tokyo, Tokyo, Japan
orcid

Research Focus & Metrics

Academic Career Length

7Years (2020-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
29 October 2024 in International Journal of Medical Informatics
Verified
Journal Article
Open access
MLMingxin Liu
Mingxin Liu
TOTsuyoshi Okuhara
Tsuyoshi Okuhara
ZDZhehao Dai
Zhehao Dai
WHWenbo Huang
Wenbo Huang
LGLin Gu
Lin Gu
TKTakahiro Kiuchi
Takahiro Kiuchi
Study aims and objectives. This study aims to evaluate the accuracy of medical knowledge in the most advanced LLMs (GPT-4o, GPT-4, Gemini 1.5 Pro, and Claude 3 Opus) as of 2024. It is the first to evaluate these LLMs using a non-English medical licensing exam. The insights from this study will guide educators, policymakers, and technical experts in the effective use of AI in medical education and clinical diagnosis. Authors inputted 790 questions from Japanese National Medical Examination into the chat windows of the LLMs to obtain responses. Two authors independently assessed the correctness. Authors analyzed the overall accuracy rates of the LLMs and compared their performance on image and non-image questions, questions of varying difficulty levels, general and clinical questions, and questions from different medical specialties. Additionally, authors examined the correlation between the number of publications and LLMs' performance in different medical specialties. GPT-4o achieved highest accuracy rate of 89.2% and outperformed the other LLMs in overall performance and each specific category. All four LLMs performed better on non-image questions than image questions, with a 10% accuracy gap. They also performed better on easy questions compared to normal and difficult ones. GPT-4o achieved a 95.0% accuracy rate on easy questions, marking it as an effective knowledge source for medical education. Four LLMs performed worst on "Gastroenterology and Hepatology" specialty. There was a positive correlation between the number of publications and LLM performance in different specialties. GPT-4o achieved an overall accuracy rate close to 90%, with 95.0% on easy questions, significantly outperforming the other LLMs. This indicates GPT-4o's potential as a knowledge source for easy questions. Image-based questions and question difficulty significantly impact LLM accuracy. "Gastroenterology and Hepatology" is the specialty with the lowest performance. The LLMs' performance across medical specialties correlates positively with the number of related publications.

EFEmi Furukawa
Emi Furukawa
TOTsuyoshi Okuhara
Tsuyoshi Okuhara
HOHiroko Okada
Hiroko Okada
Ritsuko ShirabeRitsuko Shirabe
Ritsuko Shirabe
Rie YokotaRie Yokota
Rie Yokota
TKTakahiro Kiuchi
Takahiro Kiuchi
Background: The Patient Education Materials Assessment Tool (PEMAT) systematically evaluates the understandability and actionability of patient education materials. This study aimed to develop a Japanese version of PEMAT and verify its reliability and validity. Methods: After assessing content validation, experts scored healthcare-related leaflets and videos according to PEMAT to verify inter-rater reliability. In validation testing with laypeople, the high-scoring material group (n = 800) was presented with materials that received high ratings on PEMAT, and the low-scoring material group (n = 799) with materials that received low ratings. Both groups responded to the understandability and actionability of the materials and perceived self-efficacy for the recommended actions. Results: The Japanese version of PEMAT showed strong inter-rater reliability (PEMAT-P: % agreement = 87.3, Gwet’s AC1 = 0.83. PEMAT-A/V: % agreement = 85.7, Gwet’s AC1 = 0.80). The high-scoring material group had significantly higher scores for understandability and actionability than the low-scoring material group (PEMAT-P: understandability 6.53 vs. 5.96, p < 0.001; actionability 6.04 vs. 5.49, p < 0.001; PEMAT-A/V: understandability 7.65 vs. 6.76, p < 0.001; actionability 7.40 vs. 6.36, p < 0.001). Perceived self-efficacy increased more in the high-scoring material group than in the low-scoring material group. Conclusions: Our study showed that materials rated highly on Japanese version of PEMAT were also easy for laypeople to understand and action.

15 January 2025 in JMIR Research Protocols
Verified
Journal Article
Open access
EFEmi Furukawa
Emi Furukawa
TOTsuyoshi Okuhara
Tsuyoshi Okuhara
MLMingxin Liu
Mingxin Liu
HOHiroko Okada
Hiroko Okada
TKTakahiro Kiuchi
Takahiro Kiuchi
Background: The Patient Education Materials Assessment Tool (PEMAT) is a reliable and validated instrument for assessing the understandability and actionability of patient education materials. It has been applied across diverse cultural and linguistic contexts, enabling cross-field and cross-national material quality comparisons. Accumulated evidence from studies using the PEMAT over the past decade underscores its potential impact on patient and public action. Objective: This systematic review aims to investigate how the quality of patient education materials has been assessed using the PEMAT. Methods: This review protocol follows PRISMA-P (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocols) guidelines. PubMed, MEDLINE, Cumulative Index to Nursing and Allied Health Literature (CINAHL), APA PsycInfo, and Web of Science Core Collection will be searched systematically for articles published since September 2014. Two independent reviewers will conduct the search to yield a list of relevant studies based on the inclusion and exclusion criteria. Rayyan QCRI software will be used for screening and data extraction. Results: The results will be included in the full systematic review, which is expected to start in September 2024 and be completed to be submitted for publication by early 2025. Conclusions: The findings are expected to identify the quality of materials evaluated by the PEMAT and the areas under evaluation. This review can also highlight gaps that exist in research and practice for improving the understandability and actionability of the materials, offering deeper insights into how existing materials can facilitate patient and public action. International Registered Report Identifier (IRRID): PRR1-10.2196/63489

9 July 2024 in medRxiv
Verified
Preprint
Open access
PDF file
MLMingxin Liu
Mingxin Liu
TOTsuyoshi Okuhara
Tsuyoshi Okuhara
ZDZhehao Dai
Zhehao Dai
WHWenbo Huang
Wenbo Huang
HOHiroko Okada
Hiroko Okada
TKTakahiro Kiuchi
Takahiro Kiuchi
Purpose This study aims to evaluate the accuracy of medical knowledge in the most advanced LLMs (GPT-4o, GPT-4, Gemini 1.5 Pro, and Claude 3 Opus) as of 2024. It is the first to evaluate these LLMs using a non-English medical licensing exam. The insights from this study will guide educators, policymakers, and technical experts in the effective use of AI in medical education and clinical diagnosis. Method Authors inputted 790 questions from Japanese National Medical Examination into the chat windows of the LLMs to obtain responses. Two authors independently assessed the correctness. Authors analyzed the overall accuracy rates of the LLMs and compared their performance on image and non-image questions, questions of varying difficulty levels, general and clinical questions, and questions from different medical specialties. Additionally, authors examined the correlation between the number of publications and LLMs’ performance in different medical specialties. Results GPT-4o achieved highest accuracy rate of 89.2% and outperformed the other LLMs in overall performance and each specific category. All four LLMs performed better on non-image questions than image questions, with a 10% accuracy gap. They also performed better on easy questions compared to normal and difficult ones. GPT-4o achieved a 95.0% accuracy rate on easy questions, marking it as an effective knowledge source for medical education. Four LLMs performed worst on “Gastroenterology and Hepatology” specialty. There was a positive correlation between the number of publications and LLM performance in different specialties. Conclusions GPT-4o achieved an overall accuracy rate close to 90%, with 95.0% on easy questions, significantly outperforming the other LLMs. This indicates GPT-4o’s potential as a knowledge source for easy questions. Image-based questions and question difficulty significantly impact LLM accuracy. “Gastroenterology and Hepatology” is the specialty with the lowest performance. The LLMs’ performance across medical specialties correlates positively with the number of related publications.

16 September 2023 in Clinical and Experimental Nephrology
Verified
Journal Article
Open access
PDF file
EFEmi Furukawa
Emi Furukawa
TOTsuyoshi Okuhara
Tsuyoshi Okuhara
HOHiroko Okada
Hiroko Okada
YNYuriko Nishiie
Yuriko Nishiie
TKTakahiro Kiuchi
Takahiro Kiuchi
Background: Previous studies have not fully determined whether online education materials on chronic kidney disease (CKD) for Japanese patients are easy to understand and help change their behavior. Therefore, this study quantitatively assessed the understandability and actionability of online CKD education materials. Methods: In September 2021, we searched Google and Yahoo Japan using the keywords “kidney,” “kidney disease,” “CKD,” “chronic kidney disease,” and “renal failure” to identify 538 webpages. We used the Japanese version of the Patient Education Materials Assessment Tool (PEMAT), ranging from 0 to 100%, to evaluate the understandability and actionability of webpages. We set the cutoff point to 70%. Results: Of the 186 materials included, the overall understandability and actionability were 61.5% (± 16.3%) and 38.7% (± 30.6%), respectively. The materials were highly technical in their terminology and lacked clear and concise charts and illustrations to encourage action. Compared to lifestyle modification materials on CKD overview, symptoms/signs, examination, and treatment scored significantly lower on the PEMAT. In addition, the materials produced by medical institutions and academic organizations scored significantly lower than those produced by for-profit companies. Conclusion: Medical institutions and academic organizations are encouraged to use plain language and to attach explanations of medical terms when preparing materials for patients. They are also expected to improve visual aids to promote healthy behaviors.

17 August 2023 in Nephrology
Verified
Journal Article
Open access
EFEmi Furukawa
Emi Furukawa
TOTsuyoshi Okuhara
Tsuyoshi Okuhara
HOHiroko Okada
Hiroko Okada
YFYumiko Fujitomo
Yumiko Fujitomo
MOMari Osa
Mari Osa
TKTakahiro Kiuchi
Takahiro Kiuchi
Aim: It remains undetermined whether online education videos for Japanese patients with chronic kidney disease (CKD) are easy to understand and allow adoption of behavioural changes. This study quantitatively assessed the understandability and applicability of online education videos for patients with CKD.Methods: In September 2021, we identified 200 videos on YouTube using the Japanese translations of the keywords ‘kidney,’ ‘kidney disease,’ ‘CKD,’ and ‘chronic kidney disease.’ We used the Japanese version of the Patient Education Materials Assessment Tool to evaluate the understandability and usability of webpages on a scale of 0% to 100%. The cut‐off was set at 70%.Results: A total of 54 videos were evaluated. The overall understandability and actionability of the materials were 57.6 (SD = 22.6) and 53.7 (SD = 33.4), respectively. Approximately 70% and 60% of the materials were considered insufficiently understandable and actionable, respectively. Most videos lacked a summary and had difficulty in using only everyday language. Most videos included at least one action for the audience to take. However, many failed to break down the action into sequential steps. They also lack visual aids to encourage the audience to take action.Conclusion: Our results suggest that current information on pre‐dialysis chronic kidney disease was not presented in a manner that can be applied by laypeople.