26 March 2025 in Neural Networks
28 August 2026 in SSRN Electronic Journal
Multimodal learning has been extensively exploited for brain disease diagnosis. However, integrating structural and functional magnetic resonance imaging (sMRI and fMRI) for reliable brain disease diagnosis remains a significant challenge, because unreliable cross-modal information may exist during fusion learning which leads to large mismatch between feature representation and reconstruction learning of sMRI and fMRI data. In this paper, we propose a new Reliable Multimodal Fusion Network (RMFN for short) that jointly learns spatial brain representations, temporal dynamics, and reliable cross-modal fusion. First, RMFN encodes fMRI data using Rotary Positional Embedding-enhanced Transformers to capture temporal dependencies within fMRI data. Then, RMFN represents sMRI data as region-of-interest (ROI) graphs and uses Graph Attention Networks to learn spatial relationships among brain regions. Lastly, a reliable cross-modal learning function is developed to combine both sMRI and fMRI data through adaptive weighting based on feature representation and reconstruction learning. Experiments show the utility of the proposed model. Interpretability analysis show neurobiologically significant biomarkers which are consistent with established clinical findings.
26 September 2026 in Neural Networks