The recognition of fresh tea leaf sprouts is one of the difficulties in the realization of the automated picking of fresh tea leaves. At present, the research on the detection of fresh tea leaf sprouts is based on a single variety of tea leaves for a specific period or specific place, which has no advantage for the spread, promotion, and application of the methods. To address this problem, an identification of multiple varieties of tea sprouts (IMVTS) model was proposed. First, images of three different varieties of tea (ZhongCha108 (ZC108), ZhongHuangYiHao (ZH), ZiJuan (ZJ)) were obtained, and the multiple varieties of tea (MVT) dataset for training and validating models was created. In addition, the detection effects of adding a convolutional block attention module (CBAM) or efficient channel attention (ECA) module to YOLO v7 were compared. In the detection of the MVT dataset, YOLO v7+ECA and YOLO v7+CBAM showed a higher mean average precision (mAP) than YOLO v7, with 98.82% and 98.80%, respectively. Notably, the IMVTS model had the highest AP for ZC108, ZH, and ZJ compared with the two other models, with 99.87%, 96.97%, and 99.64%, respectively. Therefore, the IMVTS model was proposed on the basic framework of the ECA and YOLO v7. To further illustrate the superiority of the model, this study also conducted a comparison test between the IMVTS model and the mainstream target detection models (YOLO v3, YOLO v5, FASTER-RCNN, and SSD) and the IMVTS model on the VOC dataset, and it is clear from the test results that the mAP of the IMVTS model is ahead of the remaining models. Concisely, the detection accuracy of the IMVTS model can meet the engineering requirements for the automatic harvesting of autumn fresh famous tea leaves, which provides a basis for the future design of detection networks for other varieties of autumn tea sprouts.