In order to solve the problem of high computational resource consumption in fabric anomaly detection, we propose a lightweight network, GH-YOLOx, which integrates ghost convolutions and hierarchical GHNetV2 backbone together to capture both local and global anomaly features. At the same time, other innovative components, such as GhostConv, dynamic convolutions, feature fusion modules, and a shared group convolution head, are applied to effectively handle multi-scale issues. Lamp pruning accelerates inference, while channel-wise knowledge distillation enhances the pruned model’s accuracy. Experiments on fabric datasets demonstrate that GH-YOLOx can effectively reduce the number of parameters while achieving a higher detection rate than other lightweight models. Overall, our solution offers a practical approach to real-time fabric anomaly detection on mobile and embedded devices.