A method for simultaneous classification of fabric material and color based on hyperspectral imaging and visual detection is proposed. Fabric material classification is performed using hyperspectral imaging (HSI) combined with a one-dimensional convolutional neural network (1D-CNN), while fabric color recognition is achieved using an red-green-blue (RGB) camera and a color classification model. Material and color features from the same fabric sample are matched to realize synchronous classification. Experiments were conducted on three fabric materials (cotton, polyester, and cotton–polyester blend) and eight colors. At a conveyor speed of 1 m/s, the sorting success rates reach 95.0% for cotton, 97.5% for polyester, and 85.0% for cotton–polyester blended fabrics. The proposed method demonstrates reliable performance for single-material fabrics and good industrial applicability for automated fabric sorting.