Infrared and visible image fusion (IVIF) aims to integrate complementary information from infrared and visible images, combining thermal target saliency with texture and structural details. Despite significant progress, existing deep fusion methods still do not fully exploit stage-specific complementary cues, and their higher-stage relation modeling may become unreliable when semantic-context guidance and structural reliability constraints are limited. To address these issues, we propose a Multi-Granularity Interaction and Semantic-Structure Rectification Network, termed MISSFusion. MISSFusion focuses on stage-aware integration of mid-level multi-granularity complementary interaction and higher-stage semantic-structure rectification. Specifically, MISSFusion introduces a Multi-Granularity Complementary Interaction Block (MCIB) to balance local details and regional context while exploiting consistency and discrepancy cues between infrared and visible features at mid-level stages. MISSFusion further develops a Semantic-Structure Rectification Attention (SSRA) module to rectify higher-stage feature relations under semantic-context guidance and structural reliability constraints, thereby improving structural consistency and suppressing unreliable relation propagation. A joint objective is designed to preserve complementary information while regularizing semantic-structure rectification. Extensive experiments on multiple IVIF benchmarks and downstream tasks suggest that MISSFusion achieves competitive performance on most evaluation metrics while maintaining a balanced trade-off among target saliency, texture preservation, and structural consistency.