Accurate matching between molecular structures and NMR spectra is an important task in automated structure elucidation. However, existing methods still face difficulties in jointly modeling multi-scale molecular topology and effectively exploiting the complementary information provided by paired ¹H and ¹³C NMR spectra. To address these limitations, we propose SpecMol-MatchNet, a multimodal matching framework that integrates a hybrid molecular graph encoder, branch-specific spectral feature learning, and residual multimodal fusion. In the molecular branch, attention-based graph interaction is combined with multi-scale neighborhood aggregation to capture structural cues at different receptive fields. In the spectral branch, branch-specific attention enhancement and joint gating are introduced to better exploit the complementary characteristics of paired ¹H and ¹³C spectra. The resulting molecular and spectral representations are integrated through a residual fusion module for final matching prediction. Experimental results on benchmark datasets demonstrate that SpecMol-MatchNet achieves consistently better overall performance than representative baseline methods.