Due to the difficulty in distinguishing transparent and white foreign fibers from seed cotton in RGB images and in order to improve the recognition ability of deep learning (DL) algorithms for white, transparent, and multi-class mixed foreign fibers with different sizes in seed cotton, this paper proposes a method of combining hyperspectral imaging technology with a multi-module joint hierarchical residue network (MJHResNet). Firstly, a series of preprocessing methods are performed on the hyperspectral image (HSI) to reduce the interference of noise. Secondly, a double-hierarchical residual (DHR) structure is designed, which can not only obtain multi-scale information, but also avoid gradient vanishing to some extent. After that, a squeeze-and-excitation network (SENet) is integrated to reduce redundant information, improve the expression of model features, and improve the accuracy of foreign fiber identification in seed cotton. Finally, by analyzing the experimental results with advanced classifiers, this method has significant advantages. The average accuracy is 98.71% and the overall accuracy is 99.28%. This method has great potential for application in the field of foreign fiber identification in seed cotton.