To address micro-break and edge-chipping defects in ferrite magnetic sheets, as well as the difficulty of balancing detection accuracy and deployment cost under complex grinding-texture interference, this paper proposes FMSD-YOLO12, an efficient and lightweight defect detection model based on YOLOv12. The proposed method follows a task-oriented design for three coupled challenges in ferrite magnetic sheet inspection, namely texture-interfered feature extraction, cross-scale feature inconsistency, and lightweight yet accurate defect localization. Specifically, a Spatially Re-weighted Convolution (SR-Conv) is introduced into the C3k2 backbone module to suppress repetitive grinding-texture noise and enhance the response contrast of subtle defect regions. A Context and Spatial Feature Calibration Network (CSFCN) is further developed to improve semantic consistency and spatial alignment during multi-scale feature fusion. In addition, a Lightweight Shared Detail-Enhanced Convolutional Detection head (LSDECD) is designed to strengthen weak-edge localization while reducing parameter redundancy through re-parameterization. Experimental results show that, with a comparable number of parameters, FMSD-YOLO12 improves mAP@50 by 2.40%, mAP@75 by 3.71%, and mAP@50-95 by 3.03% on the magnetic sheet defect dataset. These results demonstrate that the proposed model achieves a favorable balance between detection accuracy and computational efficiency for irregular defect detection under complex industrial backgrounds.