Aiming at the low qualification rate and high damage caused by the lack of identification, localization, and posture estimation of tea buds in the mechanical harvesting process of famous tea, a framework of lightweight detection + PCA-skeleton fusion posture estimation was proposed. Based on the YOLOv8n model, the StarNet backbone network was introduced to enable lightweight detection, and the ASF-YOLO multi-scale attention module was embedded to improve the feature fusion ability. Based on the detection frame, the GrabCut-Watershed fusion segmentation was employed to obtain the bud mask. Combined with PCA and skeleton extraction algorithms, the main direction deviations of bent buds and clasped leaves were solved by Bézier curve fitting, and the morphology–posture dual-factor scoring model was thereby constructed to realize the picking ranking. Compared with the original YOLOv8n model, the results showed that the detection accuracy and mAP50 of the Improved model decreased to 85.6% and 90.5%, respectively, and the recall rate increased to 81.7%. Meanwhile, the calculation load of the improved model was reduced by 23.6%, reaching 6.8 GFLOPs, indicating a significant improvement in lightweight. The morphology–posture dual-factor scoring model achieved a score of 0.88 for a single bud in vertical direction (θ ≈ 90°), a score of approximately 0.66–0.71 for buds with partially unfolded leaves and slightly bent buds, and a score of 0.48–0.53 for severely bent and overlapped buds. The results of this study have the potential to guide the picking robotic arms to preferentially pick tea buds with high adaptability and provide a reliable visual solution for low-loss and high-efficiency mechanized harvesting of famous tea in complex tea gardens.