Reconstructing high-resolution hyperspectral images (HR-HSIs) from low-resolution hyperspectral images (LR-HSIs) and high-resolution multispectral images (HR-MSIs) is an important multimodal remote sensing task for applications requiring both fine spatial details and reliable spectral characterization. However, existing Transformer-based HSI–MSI fusion methods still face difficulty in jointly preserving geometric structures and spectral continuity. Specifically, flattening two-dimensional image structures into one-dimensional token sequences tends to weaken local spatial connectivity, while standard self-attention does not explicitly model inter-band dependency, which may lead to structural distortion and spectral inconsistency in the fused results. To address these issues, this paper proposes an Adaptive Polyline Path Masked Attention Network (AdaPPMA-Net) for HSI-MSI fusion. First, an Adaptive Polyline Path Masked Attention mechanism is developed to explicitly encode horizontal and vertical geometric continuity, while a gating strategy is introduced to adaptively regulate positional constraints and suppress redundant dependencies. Second, a spectral enhancement module is embedded into the Transformer block to strengthen inter-band dependency modeling and alleviate the loss of spectral continuity during token interaction. Third, a spatial–spectral refinement (SSRefine) module is designed to recalibrate fused spatial–spectral representations, thereby improving reconstruction quality in the decoding stage. Extensive experiments on four public datasets demonstrate that AdaPPMA-Net consistently outperforms several state-of-the-art methods across multiple quantitative metrics. In particular, on the Washington DC Mall dataset, the proposed method raises PSNR by 4.2975 dB and lowers RMSE, ERGAS, and SAM by 39.03%, 39.35%, and 38.02%, respectively, compared with the strongest competing method. These results indicate that AdaPPMA-Net provides a more effective solution for high-fidelity multimodal remote sensing fusion.