Won Seok Jang

Won Seok Jang

Auto-generated profile, based on publicly available information

Won Seok Jang

Miner School of Computer and Information Sciences, University of Massachusetts Lowell, Lowell, MA, United States

Research Focus & Metrics

Academic Career Length

10Years (2017-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
7 February 2025 in Journal of Medical Internet Research
Verified
Journal Article
Open access
ZYZhichao Yang
Zhichao Yang
ZYZonghai Yao
Zonghai Yao
MTMahbuba Tasmin
Mahbuba Tasmin
PVParth Vashisht
Parth Vashisht
WJWon Seok Jang
Won Seok Jang
HYHong Yu
Hong Yu
Background: Recent advancements in artificial intelligence, such as GPT-3.5 Turbo (OpenAI) and GPT-4, have demonstrated significant potential by achieving good scores on text-only United States Medical Licensing Examination (USMLE) exams and effectively answering questions from physicians. However, the ability of these models to interpret medical images remains underexplored. Objective: This study aimed to comprehensively evaluate the performance, interpretability, and limitations of GPT-3.5 Turbo, GPT-4, and its successor, GPT-4 Vision (GPT-4V), specifically focusing on GPT-4V’s newly introduced image-understanding feature. By assessing the models on medical licensing examination questions that require image interpretation, we sought to highlight the strengths and weaknesses of GPT-4V in handling complex multimodal clinical information, thereby exposing hidden flaws and providing insights into its readiness for integration into clinical settings. Methods: This cross-sectional study tested GPT-4V, GPT-4, and ChatGPT-3.5 Turbo on a total of 227 multiple-choice questions with images from USMLE Step 1 (n=19), Step 2 clinical knowledge (n=14), Step 3 (n=18), the Diagnostic Radiology Qualifying Core Exam (DRQCE) (n=26), and AMBOSS question banks (n=150). AMBOSS provided expert-written hints and question difficulty levels. GPT-4V’s accuracy was compared with 2 state-of-the-art large language models, GPT-3.5 Turbo and GPT-4. The quality of the explanations was evaluated by choosing human preference between an explanation by GPT-4V (without hint), an explanation by an expert, or a tie, using 3 qualitative metrics: comprehensive explanation, question information, and image interpretation. To better understand GPT-4V’s explanation ability, we modified a patient case report to resemble a typical “curbside consultation” between physicians. Results: For questions with images, GPT-4V achieved an accuracy of 84.2%, 85.7%, 88.9%, and 73.1% in Step 1, Step 2 clinical knowledge, Step 3 of USMLE, and DRQCE, respectively. It outperformed GPT-3.5 Turbo (42.1%, 50%, 50%, 19.2%) and GPT-4 (63.2%, 64.3%, 66.7%, 26.9%). When GPT-4V answered correctly, its explanations were nearly as good as those provided by domain experts from AMBOSS. However, incorrect answers often had poor explanation quality: 18.2% (10/55) contained inaccurate text, 45.5% (25/55) had inference errors, and 76.3% (42/55) demonstrated image misunderstandings. With human expert assistance, GPT-4V reduced errors by an average of 40% (22/55). GPT-4V accuracy improved with hints, maintaining stable performance across difficulty levels, while medical student performance declined as difficulty increased. In a simulated curbside consultation scenario, GPT-4V required multiple specific prompts to interpret complex case data accurately. Conclusions: GPT-4V achieved high accuracy on multiple-choice questions with images, highlighting its potential in medical assessments. However, significant shortcomings were observed in the quality of explanations when questions were answered incorrectly, particularly in the interpretation of images, which could not be efficiently resolved through expert interaction. These findings reveal hidden flaws in the image interpretation capabilities of GPT-4V, underscoring the need for more comprehensive evaluations beyond multiple-choice questions before integrating GPT-4V into clinical settings.

1 March 2024 in JCO Clinical Cancer Informatics
Verified
Journal Article
Closed access
HKHyunwook Kim
Hyunwook Kim
WJWon Seok Jang
Won Seok Jang
WSWoo Seob Sim
Woo Seob Sim
HKHan Sang Kim
Han Sang Kim
JCJeong Eun Choi
Jeong Eun Choi
SSSang Joon Shin
Sang Joon Shin
PURPOSE: In artificial intelligence–based modeling, working with a limited number of patient groups is challenging. This retrospective study aimed to evaluate whether applying synthetic data generation methods to the clinical data of small patient groups can enhance the performance of prediction models. MATERIALS AND METHODS: A data set collected by the Cancer Registry Library Project from the Yonsei Cancer Center (YCC), Severance Hospital, between January 2008 and October 2020 was reviewed. Patients with colorectal cancer younger than 50 years who started their initial treatment at YCC were included. A Bayesian network–based synthesizing model was used to generate a synthetic data set, combined with the differential privacy (DP) method. RESULTS: A synthetic population of 5,005 was generated from a data set of 1,253 patients with 93 clinical features. The Hellinger distance and correlation difference metric were below 0.3 and 0.5, respectively, indicating no statistical difference. The overall survival by disease stage did not differ between the synthetic and original populations. Training with the synthetic data and validating with the original data showed the highest performances of 0.850, 0.836, and 0.790 for the Decision Tree, Random Forest, and XGBoost models, respectively. Comparison of synthetic data sets with different epsilon parameters from the original data sets showed improved performance >0.1%. For extremely small data sets, models using synthetic data outperformed those using only original data sets. The reidentification risk measures demonstrated that the epsilons between 0.1 and 100 fell below the baseline, indicating a preserved privacy state. CONCLUSION: The synthetic data generation approach enhances predictive modeling performance by maintaining statistical and clinical integrity, and simultaneously reduces privacy risks through the application of DP techniques.

7 March 2025 in JMIR Preprints
Verified
Preprint
Open access
WJWon Seok Jang
Won Seok Jang
SSSharmin Sultana
Sharmin Sultana
ZYZonghai Yao
Zonghai Yao
HTHieu Tran
Hieu Tran
ZYZhichao Yang
Zhichao Yang
HYHong Yu
Hong Yu
BACKGROUND OpenNotes allows patients to access their electronic health record (EHR) notes through online patient portals. However, EHR notes contain abundant medical jargon, which can be difficult for patients to comprehend. One way to improve comprehension is by reducing information overload and helping patients focus on the medical terms that matter most to them. OBJECTIVE In this study, we evaluated both closed-source and open-source Large Language Models (LLMs) for extracting and prioritizing medical jargon from EHR notes relevant to individual patients, leveraging prompting techniques, fine-tuning, and data augmentation. METHODS We evaluated the performance of closed-source and open-source LLMs on a dataset of 106 expert-annotated EHR notes. We tested various combinations of settings, including: i) general and structured prompts, ii) zero-shot and few-shot prompting, iii) fine-tuning, and iv) data augmentation. To enhance the extraction and prioritization capabilities of open-source models in low-resource settings, we applied data augmentation using ChatGPT and integrated a ranking technique to refine the training process. Additionally, to measure the impact of dataset size, we fine-tuned the models by incrementally increasing the size of the augmented dataset from 10 to 10,000 and tested their performance. The effectiveness of the models was assessed using 5-fold cross-validation, providing a comprehensive evaluation across various settings. We report the F1 score and Mean Reciprocal Rank (MRR) for performance evaluation. RESULTS Among the compared strategies, fine-tuning and data augmentation generally demonstrated higher performance than other approaches. Although the highest F1 score of 0.433 was achieved by GPT-4 Turbo, the highest MRR score of 0.746 was observed with Mistral7B when data augmentation was applied. Notably, by using f ine-tuning or data augmentation, open-source models were able to outperform closed-source models. Additionally, achieving the highest F1 score did not always correspond to the highest MRR score. We analyzed our experiment from several perspectives. First, few-shot prompting showed an advantage over zero-shot prompting in vanilla models. Second, when comparing general and structured prompts, each model exhibited different preferences. Third, finetuning improved zero-shot performance but sometimes degraded few-shot performance. Lastly, data augmentation yielded performance comparable to or even surpassing that of other strategies. CONCLUSIONS The evaluation of both closed-source and open-source LLMs highlighted the effectiveness of prompting strategies, fine-tuning, and data augmentation in enhancing model performance in low-resource scenarios.

1 March 2017 in European Urology Supplements
Verified
Journal Article
Closed access
WJW.S. Jang
W.S. Jang
JHJ.E. Heo
J.E. Heo
KOKyung Tak Oh
Kyung Tak Oh
MKM.S. Kim
M.S. Kim
DKD.H. Kang
D.H. Kang
YCY.D. Choi
Y.D. Choi

6 April 2025 in JMIR Preprints
Verified
Preprint
Open access
WJWon Seok Jang
Won Seok Jang
SSSharmin Sultana
Sharmin Sultana
ZYZonghai Yao
Zonghai Yao
HTHieu Tran
Hieu Tran
ZYZhichao Yang
Zhichao Yang
HYHong Yu
Hong Yu
BACKGROUND OpenNotes allows patients to access their electronic health record (EHR) notes through online patient portals. However, EHR notes contain abundant medical jargon, which can be difficult for patients to comprehend. One way to improve comprehension is by reducing information overload and helping patients focus on the medical terms that matter most to them. OBJECTIVE In this study, we evaluated both closed-source and open-source Large Language Models (LLMs) for extracting and prioritizing medical jargon from EHR notes relevant to individual patients, leveraging prompting techniques, fine-tuning, and data augmentation. METHODS We evaluated the performance of closed-source and open-source LLMs on a dataset of 106 expert-annotated EHR notes. We tested various combinations of settings, including: i) general and structured prompts, ii) zero-shot and few-shot prompting, iii) fine-tuning, and iv) data augmentation. To enhance the extraction and prioritization capabilities of open-source models in low-resource settings, we applied data augmentation using ChatGPT and integrated a ranking technique to refine the training process. Additionally, to measure the impact of dataset size, we fine-tuned the models by incrementally increasing the size of the augmented dataset from 10 to 10,000 and tested their performance. The effectiveness of the models was assessed using 5-fold cross-validation, providing a comprehensive evaluation across various settings. We report the F1 score and Mean Reciprocal Rank (MRR) for performance evaluation. RESULTS Among the compared strategies, fine-tuning and data augmentation generally demonstrated higher performance than other approaches. Although the highest F1 score of 0.433 was achieved by GPT-4 Turbo, the highest MRR score of 0.746 was observed with Mistral7B when data augmentation was applied. Notably, by using fine-tuning or data augmentation, open-source models were able to outperform closed-source models. Additionally, achieving the highest F1 score did not always correspond to the highest MRR score. We analyzed our experiment from several perspectives. First, few-shot prompting showed an advantage over zero-shot prompting in vanilla models. Second, when comparing general and structured prompts, each model exhibited different preferences. Third, fine-tuning improved zero-shot performance but sometimes degraded few-shot performance. Lastly, data augmentation yielded performance comparable to or even surpassing that of other strategies. CONCLUSIONS The evaluation of both closed-source and open-source LLMs highlighted the effectiveness of prompting strategies, fine-tuning, and data augmentation in enhancing model performance in low-resource scenarios.

6 April 2025 in JMIR AI
Verified
Journal Article
Open access
WJWon Seok Jang
Won Seok Jang
SSSharmin Sultana
Sharmin Sultana
ZYZonghai Yao
Zonghai Yao
HTHieu Tran
Hieu Tran
ZYZhichao Yang
Zhichao Yang
HYHong Yu
Hong Yu
OpenNotes allows patients to access their electronic health record (EHR) notes through online patient portals. However, EHR notes contain abundant medical jargon, which can be difficult for patients to comprehend. One way to improve comprehension is by reducing information overload and helping patients focus on the medical terms that matter most to them. This study aimed to evaluate both closed-source and open-source large language models (LLMs) for extracting and prioritizing medical jargon from EHR notes relevant to individual patients, leveraging prompting techniques, fine-tuning, and data augmentation. We evaluated the performance of closed-source and open-source LLMs on a dataset of 90 expert-annotated EHR notes. We tested various combinations of settings, including (1) general and structured prompts, (2) zero-shot and few-shot prompting, (3) fine-tuning, and (4) data augmentation. To enhance the extraction and prioritization capabilities of open-source models in low-resource settings, we applied data augmentation using GPT-4o and integrated a ranking technique to refine the training process. Additionally, to measure the impact of dataset size, we fine-tuned the models by incrementally increasing the size of the augmented dataset from 10 to 9995 and tested their performance. The effectiveness of the models was assessed using 10-fold cross-validation, providing a comprehensive evaluation across various settings. We report the F1-score and mean reciprocal rank for performance evaluation using two different string matching algorithms (relaxed string matching and Jaccard Index). We also conducted an error analysis classifying the erroneous outputs from the models. Our results show that open-source models achieved the highest performance, particularly when using fine-tuning with a gold-standard dataset. Under Jaccard Index-based string matching, DeepSeek 8B set the benchmarks with an F1-score of 0.431 (SD 0.046); similarly, BioMistral 7B showed a mean reciprocal rank of 0.577 (SD 0.109). However, under relaxed string matching, open-source models were unable to match the performance of closed-source models, even with data augmentation or fine-tuning. We analyzed our experiment from several perspectives. First, few-shot prompting did not show an advantage over zero-shot prompting in vanilla models. Second, when comparing general and structured prompts, we found that model performance could deviate largely based on prompting styles. Third, fine-tuning with a small gold-standard dataset improved performance. Finally, data augmentation yielded performance comparable to or even surpassing the fine-tuning strategy. However, it also underscored the importance of the quality of the augmented dataset. The evaluation of both closed-source and open-source LLMs highlighted the effectiveness of prompting strategies, fine-tuning, and data augmentation in enhancing model performance in low-resource scenarios.