Particleboard surface defects have a significant impact on product quality. A surface defect detection method is essential to enhancing the quality of particleboard because the conventional defect detection method has low accuracy and efficiency. This paper proposes a YOLO v5-Seg-Lab-4 (You Only Look Once v5 Segmentation-Lab-4) model based on deep learning. The model integrates object detection and semantic segmentation, which ensures real-time performance and improves the detection accuracy of the model. Firstly, YOLO v5s is used as the object detection network, and it is added into the SELayer module to improve the adaptability of the model to receptive field. Then, the Seg-Lab v3+ model is designed on the basis of DeepLab v3+. In this model, the object detection network is utilized as the backbone network of feature extraction, and the expansion rate of atrus convolution is reduced to the computational complexity of the model. The channel attention mechanism is added onto the feature fusion module, for the purpose of enhancing the feature characterization capabilities of the network algorithm as well as realizing the rapid and accurate detection of lightweight networks and small objects. Experimental results indicate that the proposed YOLO v5-Seg-Lab-4 model has mAP (Mean Average Precision) and mIoU (Mean Intersection over Union) of 93.20% and 76.63%, with a recognition efficiency of 56.02 fps. Finally, a case study of the Huizhou particleboard factory inspection is carried out to demonstrate the tiny detection accuracy and real-time performance of this proposed method, and the missed detection rate of surface defects of particleboard is less than 1.8%.