Early detection of hidden maize weevil (Sitophilus zeamais) infestation in wheat is difficult, and moisture variation can obscure weak near-infrared spectral responses. This study investigated wheat in three nominal moisture groups designated as 13%, 16% and 19%. Hyperspectral data and corresponding three-channel pseudocolor images were acquired from healthy and infested kernels at four operationally defined sampling times to construct single- and mixed-moisture datasets. An Asymmetric Dual-Stream Masked Autoencoder Fusion Network (ADS-MAFNet) was developed by combining a CNN–Transformer spectral branch with masked autoencoder (MAE) pretraining and a 2D-CNN image branch for binary classification. Classification performance varied across the nominal moisture groups, with most spectral deep learning models performing less well in the 19% group. Within the investigated datasets, ADS-MAFNet achieved higher mean accuracy and F1-score than the evaluated single-branch neural models in the nominal 19% and mixed-moisture groups. Its mixed-moisture accuracy, precision, recall, and F1-score were 79.40%, 75.27%, 84.28%, and 79.44%, respectively. However, PLS-DA achieved higher mean accuracy and F1-score than ADS-MAFNet under all four evaluated moisture conditions. Ablation comparisons showed larger ten-run mean differences for branch removal than for removal of MAE pretraining, but a consistent advantage over IMG-only and a consistent pretraining benefit were not established. Classification performance was generally higher at the two later sampling times than at the two earlier sampling times. These results support further investigation of hyperspectral image fusion within the represented laboratory conditions; transfer to unseen moisture levels and independent batches remains untested.