To enhance the precision of cottonseed impurity detection and address issues such as high miss-detection rates and suboptimal performance, this paper introduces an improved YOLOv11 algorithm, termed YOLOv11-SMS. Initially, the algorithm integrates a local self-attention mechanism (LRSA) to design the C2PSA-SL module, which augments the model’s ability to learn local information while maintaining global feature awareness. Furthermore, the feature extraction stage and the network head incorporate a multi-branch reparameterized convolution (MBRConv) module, enhancing feature extraction capabilities while preserving the model’s lightweight properties. Lastly, a spatial adaptive modulation (SAFM) module is introduced to optimize the detection of small targets. Experimental results demonstrate that YOLOv11-SMS outperforms the baseline model, with mAP@_50–95 increasing from 79.42% to 82.49%, an improvement of 3.07 percentage points. The average mIOU increased from 90.98% to 94.18%, representing a 3.2 percentage point improvement. Moreover, the model achieves an impressive real-time inference speed of 178.63 frames per second (FPS), effectively balancing detection accuracy and speed, offering an efficient and precise solution for cottonseed impurity detection.