As an indispensable technology in agricultural production, crop breeding, and seed inspection, the effective classification of rice seed varieties cannot only eliminate counterfeit and inferior seeds but also facilitate the careful selection and grading of rice seeds. Traditional methods often suffer from cumbersome operations, seed damage, and high costs. To address these issues, we employ hyperspectral imaging and deep learning to achieve simple, efficient, and non-destructive rice seed classification. Specifically, a mixed domain-based gradient attention module is proposed, which utilizes a pyramid structure to encode gradient weights obtained from five classic image operators. This facilitates the effective extraction of detailed features while reducing noise and irrelevant information. To support our research, two hyperspectral image datasets of rice seeds generated by a data augmentation strategy are constructed and will be released. A series of performance evaluations demonstrates the feasibility of our methods for rice seed variety classification.