Accurate postharvest quality assessment of specialty grains remains challenging due to complex quality variations and the limited deployment of multimodal sensing systems. RGB imaging provides efficient external phenotypic information, whereas visible–near-infrared (VNIR) sensing provides complementary wavelength-dependent reflectance information. However, the high cost and complexity of spectral acquisition restrict the practical application of multimodal models. This study proposes an RGB–VNIR cross-modal decision residual distillation framework for non-destructive quality recognition of Canadian wild rice kernels. The proposed Decision Spectral Residual Extractor (DSRE) models VNIR information as decision-level residual knowledge relative to RGB predictions, while the Cross-modal Residual Reliability Adaptation (CRRA) selectively transfers transferable spectral knowledge according to residual utility and recoverability. Experiments on eight quality categories of Canadian wild rice kernels demonstrated the effectiveness of the proposed framework. On the paired RGB–VNIR test set (n = 384), the DSRE-based multimodal teacher achieved 97.40% Accuracy, compared with 94.53% for the CRRA RGB-only student on the same samples, confirming the complementary contribution of VNIR information. In a separate evaluation on the held-out RGB-only test set (n = 1570), the CRRA student achieved 98.60% Accuracy and 98.62% Macro-F1, indicating effective multimodal knowledge transfer without VNIR acquisition during inference. The proposed framework establishes practical multimodal learning with a unimodal deployment paradigm, providing a promising solution for cost-effective and non-destructive quality evaluation of specialty grains.