To address the challenges associated with low-contrast defect representation, loss of small-scale fine details, and blurred boundaries in infrared thermographic testing of defects in carbon fiber-reinforced composites, this paper proposes band-guided hierarchical fusion and decoupled segmentation (BHF-DS), a lightweight infrared instance segmentation network designed for efficient and accurate defect detection and instance segmentation. The proposed network focuses on enhancing weak defect features, improving multi-scale detail fusion, and reducing computational redundancy in the prediction head. Specifically, a band-adaptive Haar stem is introduced to model and reweight features from low- and high-frequency subbands to enhance defect characterization. An adaptive weighted feature concatenation module is employed to facilitate cross-layer interaction between shallow details and deep semantic features. Additionally, a channel-aligned partial-convolution decoupled segmentation head is constructed to improve the defect localization and mask prediction quality while minimizing the model complexity. The proposed method was validated using infrared thermograms collected from self-fabricated carbon fiber-reinforced composites, specimens with artificially induced defects. The defect types included simulated bubbles, simulated delamination, and flat-bottom holes of various sizes and buried depths. Experimental results demonstrate that BHF-DS achieves AP50(Box), AP50-95(Box), AP50(Seg), and AP50-95(Seg) scores of 0.9295, 0.4964, 0.9102, and 0.4501, respectively, on the fixed test set, while attaining an inference speed of 78.32 FPS on the NVIDIA A10 GPU platform with 9.71 GFLOPs and 2.297 M parameters. Overall, BHF-DS improves the detection and instance segmentation accuracy of low-contrast defects in infrared thermograms of carbon fiber-reinforced composites. Meanwhile, BHF-DS maintains a lightweight model design and efficient inference performance, demonstrating its potential for automated infrared thermographic inspection with a favorable balance between accuracy and efficiency.