Rapid and non-destructive detection methods for the withering degree of fresh tea leaves are crucial for ensuring high-quality tea production. Therefore, this study proposes a fresh tea withering degree detection model based on image classification confidence. The moisture percentage of fresh tea leaves is calculated by developing a weighted method that combines confidence levels and moisture labels, and the degree of withering is ultimately determined by incorporating the standard for wilted moisture content. To enhance the feature extraction ability and classification accuracy of the model, we introduce the Receptive-Field Attention Convolution (RFAConv) and Cross-Stage Feature Fusion Coordinate Attention (C2f_CA) modules. The experimental results demonstrate that the proposed model achieves a classification accuracy of 92.7%. Compared with the initial model, the detection accuracy was improved by 0.156. In evaluating the predictive performance of the model for moisture content, the correlation coefficients (Rp), root mean square error (RMSEP), and relative standard deviation (RPD) of category 1 in the test set were 0.9983, 0.006278, and 39.2513, respectively, and all performance were significantly better than PLS and CNN methods. This method enables accurate and rapid detection of tea leaf withering, providing crucial technical support for online determination during processing.