Infrared–visible image fusion must reconcile different sensor roles: infrared imagery provides illumination-robust thermal evidence, whereas visible imagery supplies richer but illumination-sensitive structures and textures. Existing vision–language fusion methods show that textual context can guide reconstruction, but they rarely examine where language should enter a modality-specific network. We propose VBT-Fusion, a branch-selective language-conditioning framework for infrared–visible fusion. A precomputed language sequence interacts only with the visible branch, while the infrared representation follows an independent text-free path. In three Restormer-CA stages, text tokens query projected visible tokens, and the attended response is pooled and normalized to generate a global language-conditioned weighting vector for visible-feature calibration. Stage-1 modality features form an early fusion representation that preserves local structure and fine detail, whereas Stage-3 features form a deep representation with stronger contextual and thermal responses. A fusion convolution combines the two representations before a four-level U-Net reconstructs the fused image. Experiments on MSRS, M3FD, and RoadScene demonstrate strong in-domain performance and competitive cross-dataset generalization. VBT-Fusion ranks first on all seven reported MSRS metrics and preserves information richness and high-frequency detail on RoadScene, although several source-fidelity measures reveal a complementary trade-off. Ablation results support visible-branch language conditioning, cross-depth aggregation, and hierarchical reconstruction.