Hyungyung Lee

Hyungyung Lee

Auto-generated profile, based on publicly available information

Hyungyung Lee

Graduate School of AI, KAIST, Daejeon, South Korea

Research Focus & Metrics

Academic Career Length

6Years (2021-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
19 September 2022 in IEEE Journal of Biomedical and Health Informatics
Verified
Journal Article
Closed access
JMJong Hak Moon
Jong Hak Moon
HLHyungyung Lee
Hyungyung Lee
WSWoncheol Shin
Woncheol Shin
YKYoung-Hak Kim
Young-Hak Kim
ECEdward Choi
Edward Choi
Recently a number of studies demonstrated impressive performance on diverse vision-language multi-modal tasks such as image captioning and visual question answering by extending the BERT architecture with multi-modal pre-training objectives. In this work we explore a broad set of multi-modal representation learning tasks in the medical domain, specifically using radiology images and the unstructured report. We propose Medical Vision Language Learner (MedViLL), which adopts a BERT-based architecture combined with a novel multi-modal attention masking scheme to maximize generalization performance for both vision-language understanding tasks (diagnosis classification, medical image-report retrieval, medical visual question answering) and vision-language generation task (radiology report generation). By statistically and rigorously evaluating the proposed model on four downstream tasks with three radiographic image-report datasets (MIMIC-CXR, Open-I, and VQA-RAD), we empirically demonstrate the superior downstream task performance of MedViLL against various baselines, including task-specific architectures. The source code is publicly available at: https://github.com/SuperSupermoon/MedViLL

29 June 2021 in JMIR Preprints
Verified
Preprint
Open access
JMJong Hak Moon
Jong Hak Moon
HLHyungyung Lee
Hyungyung Lee
WSWoncheol Shin
Woncheol Shin
YKYoung-Hak Kim
Young-Hak Kim
ECEdward Choi
Edward Choi
BACKGROUND Recently a number of studies demonstrated impressive performance on diverse vision-language multi-modal tasks such as image captioning and visual question answering by extending the self-attention based Transformer architecture with multi-modal pre-training objectives. Despite its huge potential, vision-language multi-modal pre-training in the medical domain has only recently received attention, only demonstrated improved diagnosis accuracy of vision-language pre-trained models. OBJECTIVE In this work we explore a broad set of multi-modal representation learning tasks in the medical domain, specifically using radiology images and the unstructured report. METHODS We propose Medical Vision Language Learner (MedViLL) which adopts a Transformer-based architecture combined with a novel multi-modal attention masking scheme to maximize generalization performance for both vision-language understanding tasks (disease classification, medical image-report retrieval, and medical visual question answering) and vision-language generation task (radiology report generation). RESULTS By rigorously evaluating the proposed model on four downstream tasks with three radiographic image-text datasets (MIMIC-CXR, Open-I, and VQA-RAD), we empirically demonstrate the superior downstream task performance and generality of MedViLL against various baselines including task-specific architectures. In addition, we qualitatively analyze MedViLL by showing the results of retrieved image-report pair, the attention map visualization, and generated reports. CONCLUSIONS Our proposed multi-modal pre-training model MedViLL could flexibly adapt to multiple downstream tasks of vision-language understanding and generation with a novel self-attention scheme. We believe that our approach can provide the basis for a wide range of interpretations of vision-language multi-modal in the medical domain.