Deep learning for defect detection has become prevalent to improve fabric production and efficiency and quality. However, fabric defect detection faces challenges such as low recognition ratio, suboptimal classification performance, poor detection inefficiency, and high model complexity. To address these issues, an end-to-end semantic segmentation network is proposed employing an efficient encoder-decoder structure, denoted as Feature Pyramid-Deeplab (FP-Deeplab). The improvements involves enhancing the backbone network by improving the MobileNetV3 network for superior performance, a novel Atrous Spatial Pyramid Pooling with Dilated Strip Pooling (ASPP-DSP) module which combines strip pooling, dilated convolution and ASPP, to ensure an expanded receptive field and the capability to gather distant contextual information. Additionally, a Feature Pyramid module (FP module) is proposed to integrate multiscale features at various stages more efficiently. The incorporating of depth-wise separable convolution in FP-Deeplab enables significant parameter and computational cost reduction, catering to online detection requirements. Experimental results certify the superiority of FP-Deeplab over classical and recent segmentation models. Comparative analysis demonstrates higher segmentation accuracy and reduced parameter quantity. Specifically, compared to the benchmark Deeplabv3+ model with MobileV2 as the backbone, FP-Deeplab achieves a notable increase in segmentation accuracy (F1 score and MIoU) by 4.26% and 5.81%, respectively. Moreover, the model parameters (params) are only one-fifth of the original model, indicating the efficiency and effectiveness of our proposed approach.