Seed varietal purity and physiological viability are critical determinants of crop yield and quality. However, non-destructive assessment faces significant challenges in fine-grained variety discrimination and the perception of internal defects. This study proposes S3-Net, an AI-driven multimodal sensing framework that integrates vision–language alignment with dual-spectral sensor fusion for autonomous seed quality evaluation. We introduce a Knowledge–Vision Alignment (KVA) module that incorporates encyclopedic morphological descriptions to guide feature learning, significantly enhancing few-shot generalization. Complementarily, a Dual-Spectral Fusion (DSF) module combines high-resolution RGB textures with penetrative Short-Wave Infrared (SWIR) sensing to jointly characterize external and internal traits. Experimental results on a custom multimodal dataset of 6000 samples across 12 crop categories demonstrate that S3-Net achieves 96.9% accuracy for species identification and 95.8% for viability detection. Notably, S3-Net outperforms ResNet-50 by 40.3% in extreme 1-shot scenarios. With a stable inference throughput of 95 fps, the system meets the high-throughput demands of industrial-scale applications, providing a robust and efficient solution for intelligent agricultural phenotyping.