5 January 2024 in IEEE Journal of Biomedical and Health Informatics
Recently, automatic radiology report generation, which targets to generate multiple sentences that can accurately describe medical observations for given X-ray images, has gained increasing attention. Existing methods commonly employ the attention mechanism for accurate word generation. However, such attention-based methods fail to leverage useful image-level global features, thereby limiting the model's reasoning ability. To tackle this challenge, we propose two-stage global enhancement layers to facilitate the Transformer to generate more reliable reports from a global perspective. Specifically, the $1^{st}$ Global Enhancement Layer ( $1^{st}$ GEL) is designed to capture the global visual context features by establishing the relationships between image-level global features and previously generated words. The $2^{nd}$ Global Enhancement Layer ( $2^{nd}$ GEL) is devised to capture the region-global level features by building the relationships between image-level global features and region-level information. The experiments demonstrate that by integrating the aforementioned two-stage global enhancement layers into the Transformer model, our proposal achieves state-of-the-art (SOTA) performance on various Natural Language Generation (NLG) evaluation metrics. Further Clinical Efficacy (CE) evaluations also validate that our proposal is able to predict more critical information. The Code will be available at https://github.com/SKD-HPC/TSGET .
27 November 2024 in IEEE Transactions on Medical Imaging
Radiology report generation that aims to accurately describe medical findings for given images, is pivotal in contemporary computer-aided diagnosis. Recently, despite considerable progress, current radiology report generation models still struggled to achieve consistent quality across difficult and easy samples, which dramatically impacts their clinical value. To solve this problem, we explore the difficult samples mining in radiology report generation and propose the Linear Hybrid-Reward based Reinforced Focal Learning (LHR-RFL) to effectively guide the model to allocate more attention towards some difficult samples, thereby enhancing its overall performance in both general and intricate scenarios. In implementation, we first propose the Linear Hybrid-Reward (LHR) module to better quantify the learning difficulty, which employs a linear weighting scheme that assigns varying weights to three representative Natural Language Generation (NLG) evaluation metrics. Then, we propose the Reinforced Focal Learning (RFL) to adaptively adjust the contributions of difficult samples during training, thereby augmenting their impact on model optimization. The experimental results demonstrate that our proposed LHR-RFL improves the performance of the base model across all NLG evaluation metrics, achieving an average performance improvement of 20.9% and 13.2% on IU X-ray and MIMIC-CXR datasets, respectively. Further analysis also proves that our LHR-RFL can dramatically improve the quality of reports for difficult samples. The source code will be available at https://github.com/SKD-HPC/LHR-RFL.
12 September 2025 in Neural Networks
20 January 2022 in Soft Computing