Boron-doped diamond thin films exhibit extensive applications in chemical sensing, in which the performance could be further enhanced by nano-structuring of the surfaces. In order to discover the relationship between diamond nanostructures and properties, this paper is dedicated to deep learning target detection methods. However, great challenges, such as noise, unclear target boundaries, and mutual occlusion between targets, are inevitable during the target detection of nanostructures. To tackle these challenges, DWS-YOLOv8 (DCN + WIoU + SA + YOLOv8n) is introduced to optimize the YOLOv8n model for the detection of diamond nanostructures. A deformable convolutional C2f (DCN_C2f) module is integrated into the backbone network, as is a shuffling attention (SA) mechanism, for adaptively tuning the perceptual field of the network and reducing the effect of noise. Finally, Wise-IoU (WIoU)v3 is utilized as a bounding box regression loss to enhance the model’s ability to localize diamond nanostructures. Compared to YOLOv8n, a 9.4% higher detection accuracy is achieved for the present model with reduced computational complexity. Additionally, the enhancement of precision (P), recall (R), mAP@0.5, and mAP@0.5:0.95 is demonstrated, which validates the effectiveness of the present DWS-YOLOv8 method. These methods provide effective support for the subsequent understanding and customization of the properties of surface nanostructures.