As the raw material for tea making, the quality of fresh tea leaves directly affects the quality of finished tea. Traditional manual sorting and machine sorting struggle to meet the requirements for high-quality tea processing. Based on machine vision and deep learning, intelligent grading technology has been applied to the automated sorting of fresh tea leaves. However, when faced with machine-picked tea leaves, the characteristics of complex morphology, small target recognition size, and dense spatial distribution can interfere with accurate category recognition, which in turn limits classification accuracy and consistency. Therefore, we propose an enhanced YOLOv12 detection framework that integrates three key modules—C3k2_EMA, A2C2f_DYT, and RFAConv—to strengthen the model’s ability to capture delicate tea bud features, thereby improving detection accuracy and robustness. Experimental results demonstrate that the proposed method achieves precision, recall, and mAP@0.5 of 81.2%, 90.6%, and 92.7% in premium tea recognition, effectively supporting intelligent and efficient tea harvesting and sorting operations. This study addresses the challenges of subtle fine-grained differences, small object sizes, variable morphology, and complex background interference in premium tea bud images. The proposed model not only achieves high accuracy and robustness in fine-grained tea bud detection but also provides technical feasibility for intelligent fresh tea leaves classification and production monitoring.