A field-based apple detection and grading device was developed and used to detect and grade apples in the field using a deep learning framework. Four features were selected for apple grading, namely, size, color, shape, and surface defects, and detection algorithms were designed to discriminate between the four features using machine vision and other methods. Then, the four apple features were fused, and a support vector machine (SVM) was used for infield apple grading into three grades: first-grade fruit, second-grade fruit, and other-grade fruit. The results showed that for a single index, the accuracy of detecting the apple size, the fruit shape, the color, and the surface defects, were 99.04%, 97.71%, 98%, and 95.85%. The grading accuracies for the first-grade fruit, second-grade fruit, other-grade fruit, and the average grading accuracy based on multiple features were 94.55%, 95.71%, 100%, and 95.49%, respectively. The field experiment showed that the average grading accuracy was 94.12% when the feeding interval of the apples was less than 1.5 s and the walking speed did not exceed 0.5 m/s, meeting the accuracy requirements of field-based apple grading.