Walnuts with their shells are a popular agricultural product in China. However, mildew from growth can sometimes be processed into foods. It is difficult to visually determine which walnuts have mildew without breaking the shells. A non-destructive method for detecting walnuts with mildew was studied by combining spectral data with image information. A total of 120 “Lüling” walnuts with shells were used for the mildew experiment. The characteristics of the spectral data from six surfaces of all samples were collected in the range of 370–1042 nm on days 0, 15, and 30. The spectrum was pretreated using SNV, and the feature bands were extracted using PCA and modeled using a support vector machine (SVM). The results show that the overall classification accuracy was 93%, with an of accuracy of 100% for INEN walnuts (normal internally and externally). The accuracy for IMEM walnuts (mildew internally and externally) reached 87.29%. There was an accuracy of 78.6% for IMEN walnuts (mildew internally and normal externally). The non-destructive detection of mildewed walnuts can be undertaken using hyperspectral imaging technology, which provides a new technique for exploring the mechanisms of walnuts with mildew.