12 February 2024 in IEEE Transactions on Artificial Intelligence
The “Impression” section of a radiology report is a critical basis for communication between radiologists and other physicians. Typically written by radiologists, this part is derived from the “Findings” section, which can be laborious and error-prone. Although deep-learning based models, such as BERT, have achieved promising results in Automatic Impression Generation (AIG), such models often require substantial amounts of medical data and have poor generalization performance. Recently, Large Language Models (LLMs) like ChatGPT have shown strong generalization capabilities and performance, but their performance in specific domains, such as radiology, remains under-investigated and potentially limited. To address this limitation, we propose ImpressionGPT, leveraging the contextual learning capabilities of LLMs through our dynamic prompt and iterative optimization algorithm to accomplish the AIG task. ImpressionGPT initially employs a small amount of domain-specific data to create a dynamic prompt, extracting contextual semantic information closely related to the test data. Subsequently, the iterative optimization algorithm automatically evaluates the output of LLMs and provides optimization suggestions, continuously refining the output results. The proposed ImpressionGPT model achieves superior performance of AIG task on both MIMIC-CXR and OpenI datasets without requiring additional training data or fine-tuning the LLMs. This work presents a paradigm for localizing LLMs that can be applied in a wide range of similar application scenarios, bridging the gap between general-purpose LLMs and the specific language processing needs of various domains.
14 March 2022 in IEEE Transactions on Neural Networks and Learning Systems
Graph neural networks (GNNs) have received increasing interest in the medical imaging field given their powerful graph embedding ability to characterize the non-Euclidean structure of brain networks based on magnetic resonance imaging (MRI) data. However, previous studies are largely node-centralized and ignore edge features for graph classification tasks, resulting in moderate performance of graph classification accuracy. Moreover, the generalizability of GNN model is still far from satisfactory in brain disorder [e.g., autism spectrum disorder (ASD)] identification due to considerable individual differences in symptoms among patients as well as data heterogeneity among different sites. In order to address the above limitations, this study proposes a novel adversarial learning-based node-edge graph attention network (AL-NEGAT) for ASD identification based on multimodal MRI data. First, both node and edge features are modeled based on structural and functional MRI data to leverage complementary brain information and preserved in the constructed weighted adjacent matrix for individuals through the attention mechanism in the proposed NEGAT. Second, two AL methods are employed to improve the generalizability of NEGAT. Finally, a gradient-based saliency map strategy is utilized for model interpretation to identify important brain regions and connections contributing to the classification. Experimental results based on the public Autism Brain Imaging Data Exchange I (ABIDE I) data demonstrate that the proposed framework achieves a classification accuracy of 74.7% between ASD and typical developing (TD) groups based on 1007 subjects across 17 different sites and outperforms the state-of-the-art methods, indicating satisfying classification ability and generalizability of the proposed AL-NEGAT model. Our work provides a powerful tool for brain disorder identification.
17 June 2025 in IEEE Transactions on Neural Networks and Learning Systems
Large language models (LLMs) such as ChatGPT have recently demonstrated significant potential in mathematical abilities, providing a valuable reasoning paradigm consistent with human natural language. However, LLMs currently have difficulty in bridging perception, language understanding, and reasoning (PLR) capabilities due to incompatibility of the underlying information flow among them, making their reasoning ability not fully elicited and challenging to accomplish complicated reasoning tasks autonomously. To resolve the above problem, a novel method called ChatABL is proposed by integrating LLMs into an abductive learning (ABL) framework, capable of unifying the three abilities effectively in a more user-friendly and understandable manner. Initially, the proposed method uses LLMs to correct the incomplete logical facts for optimizing the perception module, by summarizing and reorganizing domain knowledge represented in natural language format. Then, the perception module also provides necessary logical reasoning materials for feeding LLMs. Finally, these parts are integrated into a dynamic closed-loop system by introducing the feedback form and automatic learning strategies to mutually promote their performance. As a testbed, the variable-length handwritten equation decipherment (HED), an abstract expression of the Mayan calendar decoding, is used to demonstrate that ChatABL has reasoning ability beyond most existing state-of-the-art methods, which has been well-supported by comparative studies. To the best of authors’ knowledge, the proposed ChatABL is the first attempt to explore a possible and novel avenue to approaching human-level cognitive ability via natural language interaction by means of ChatGPT.
16 January 2020 in Brain Imaging and Behavior
Mapping the relation between cortical convolution and structural/functional brain architectures could provide deep insights into the mechanisms of brain development, evolution and diseases. In our previous studies, we found a unique gyral folding pattern, termed a 3-hinge, which was defined as the conjunction of three gyral crests. The uniqueness of the 3-hinge was evidenced by its thicker cortex and stronger fiber connections than other gyral regions. However, the role that 3-hinges play in cortico-cortical connective architecture remains unclear. To this end, we conducted MRI studies by constructing structural cortico-cortical connective networks based on a fine-granular cortical parcellation, the parcels of which were automatically labeled as 3-hinge, 2-hinge (ordinary gyrus) or sulcus. On human brains, 3-hinges possess significantly higher degrees, strengths and betweennesses than 2-hinges, suggesting that 3-hinges could serve more like hubs in the cortico-cortical connective network. This hypothesis gains supports from human functional network analyses, in which 3-hinges are involved in more global functional networks than ordinary gyri. In addition, 3-hinges could serve as ‘connector’ hubs rather than ‘provincial’ hubs and they account for a dominant proportion of nodes in the high-level ‘backbone’ of the network. These structural results are reproduced on chimpanzee and macaque brains, while the roles of 3-hinges as hubs become more pronounced in higher order primates. Our new findings could provide a new window to the relation between cortical convolution, anatomical connection and brain function.
1 March 2026 in IEEE Transactions on Biomedical Engineering
Achieving clinical level performance and widespread deployment for generating radiology impressions encounters a giant challenge for conventional artificial intelligence models tailored to specific diseases and organs. Concurrent with the increasing accessibility of radiology reports and advancements in modern general AI techniques, the emergence and potential of deployable radiology AI exploration have been bolstered. Here, we present ChatRadio-Valuer, the first general radiology diagnosis large language model for localized deployment within hospitals and being close to clinical use for multi-institution and multi-system diseases. ChatRadio-Valuer achieved 15 state-of-the-art results across five human systems and six institutions in clinical-level events (n = 332,673) through rigorous and full-spectrum assessment, including engineering metrics, clinical validation, and efficiency evaluation. Notably, it exceeded OpenAI's GPT-3.5 and GPT-4 models, achieving superior performance in comprehensive disease diagnosis compared to the average level of radiology experts. Besides, ChatRadio-Valuer supports zero-shot transfer learning, greatly boosting its effectiveness as a radiology assistant, while ensuring adherence to privacy standards and being readily utilized for large-scale patient populations. Our expeditions suggest the development of localized LLMs would become an imperative avenue in hospital applications.