Defect detection holds significant importance in improving the overall quality of fabric manufacturing. To improve the effectiveness and accuracy of fabric defect detection, we propose the PRC-Light YOLO model for fabric defect detection and establish a detection system. Firstly, we have improved YOLOv7 by integrating new convolution operators into the Extended-Efficient Layer Aggregation Network for optimized feature extraction, reducing computations while capturing spatial features effectively. Secondly, to enhance the performance of the feature fusion network, we use Receptive Field Block as the feature pyramid of YOLOv7 and introduce Content-Aware ReAssembly of FEatures as upsampling operators for PRC-Light YOLO. By generating real-time adaptive convolution kernels, this module extends the receptive field, thereby gathering vital information from contexts with richer content. To further optimize the efficiency of model training, we apply the HardSwish activation function. Additionally, the bounding box loss function adopts the Wise-IOU v3, which incorporates a dynamic non-monotonic focusing mechanism that mitigates adverse gradients from low-quality instances. Finally, in order to enhance the PRC-Light YOLO model’s generalization ability, we apply data augmentation techniques to the fabric dataset. In comparison to the YOLOv7 model, multiple experiments indicate that our proposed fabric defect detection model exhibits a decrease of 18.03% in model parameters and 20.53% in computational load. At the same time, it has a notable 7.6% improvement in mAP.