Automatic defect segmentation of atmospheric plasma sprayed (APS) thermal barrier coatings (TBCs) from scanning electron microscopy (SEM) images is essential for quantitative microstructural assessment; however, existing instance segmentation models often exhibit insufficient multi-scale representation and excessive computational cost. To address these limitations, we propose an improved YOLO11-seg–based framework that jointly optimizes accuracy and efficiency through two architectural enhancements: a Boundary-aware Recalibrated Bidirectional Pyramid Network (BRB-FPN) for strengthened cross-scale feature fusion and precise contour localization, and an Efficient Shared-Convolution Segmentation Head (ESCH) for reducing parameters and FLOPs while accelerating inference. Without introducing additional training strategies, the proposed method achieves consistent performance gains on a self-constructed APS–TBC SEM defect dataset, improving Mask mAP@50 by 6.9%, Mask mAP@50:95 by 4.4%, and recall by 5.5% over the baseline, while simultaneously lowering computational complexity and increasing FPS, and outperforming several mainstream segmentation approaches. These results demonstrate that the proposed framework provides an accurate and computationally efficient solution for industrial microscopic defect inspection.