The detection of insulator defects plays a critical role in ensuring the stable operation of power transmission and distribution networks. In response to the problems of low speed, low accuracy, and difficulty in deploying to the embedded terminals in existing insulator defect detection, this paper proposes a lightweight insulator defect detection method based on improved convolutional neural network, named LIDD-Net. Firstly, a novel multi-scale feature fusion strategy, called Direct Feature Fusion (DFF), is proposed to significantly reduce model parameters while maintaining high detection accuracy in complex backgrounds, along with improvements to the two lightweight modules, LightUnit1 and LightUnit2. Secondly, to enhance sample selection accuracy, a new matching strategy, atssMatcher, is proposed. Finally, by incorporating auxiliary training modules, including the Small Object Detector (SO Detector) and the Large Object Detector (LO Detector ), the model effectively improves detection accuracy for targets of different sizes. Experimental results demonstrated that, with only 4.3M parameters, LIDD-Net achieved a mean Average Precision (mAP) of 89.3% on the CPLID-220 dataset, with a defect category Average Precision (AP) of 97.8%. Additionally, LIDD-Net reaches 99.0 frames/s, meeting the real-time detection requirements of insulator defects.