With the development of industrialization, the demand for high-performance metal materials has increased, and copper and its alloys have been widely used. The microstructure of these materials significantly affects their performance. To address the issues of subjectivity, low efficiency, and limited quantitative capability in traditional metallographic analysis methods, this paper proposes a deep learning-based approach for segmenting the second phase in Cu-Fe alloys. The method is built upon the YOLO11 framework and incorporates a series of structural enhancements tailored to the characteristics of the secondary-phase microstructure, aiming to improve the model’s detection accuracy and segmentation performance. Specifically, the EIEM module enhances the C3K2 structure to improve edge perception; the CSPSA module is optimized into C2CGA to strengthen multi-scale feature representation; and the RepGFPN and DySample techniques are integrated to construct the GDFPN neck network. Experimental results on the Cu-Fe alloy metallographic image dataset demonstrate that YOLO11 outperforms mainstream semantic segmentation models such as U-Net and DeepLabV3+ in terms of mAP (85.5%), inference speed (208 FPS), and model complexity (10.2 GFLOPs). The improved YOLO11 model achieves an mAP of 89.0%, a precision of 84.6%, and a recall of 81.0% on this dataset, showing significant performance improvements while effectively balancing inference speed and model complexity. Additionally, a quantitative analysis software system for secondary phase uniformity based on this model provides strong technical support for automated metallographic image analysis and demonstrates broad application prospects in materials science research and industrial quality control.