Hyperspectral imaging for seed vigor detection faces the challenges of handling high-dimensional spectral data, information loss after dimensionality reduction, and low feature differentiation between vigor levels. To address the above issues, this study proposes an improved dynamic optimize MDS (DO-MDS) dimensionality reduction algorithm based on multidimensional scaling transformation. DO-MDS better preserves key features between samples during dimensionality reduction. Secondly, a dual-stream spectral collaborative attention (DSCA) module is proposed. The DSCA module adopts a dual-modal fusion approach combining global feature capture and local feature enhancement, deepening the characterization capability of spectral features. This study selected commonly used rice seed varieties in Zhejiang Province and constructed three individual spectral datasets and a mixed dataset through aging, spectral acquisition, and germination experiments. The experiments involved using the DO-MDS processed datasets with a convolutional neural network embedded with the DSCA attention module, and the results demonstrate vigor discrimination accuracy rates of 93.85%, 93.4%, and 96.23% for the Chunyou 83, Zhongzao 39, and Zhongzu 53 datasets, respectively, achieving 94.8% for the mixed dataset. This study provides effective strategies for spectral dimensionality reduction in hyperspectral seed vigor detection and enhances the differentiation of spectral information for seeds with similar vigor levels.