The recognition of coal and gangue is the premise and foundation of coal gangue intelligent sorting. Adaptive boosting (AdaBoost) algorithm-based coal gangue identification has not been studied in depth. This paper proposed a coal gangue image recognition algorithm and a strong classifier based on the AdaBoost algorithm with a genetic algorithm (GA)-optimized support vector machine (SVM). One thousand coal gangue images were collected on-site and expanded to five thousand via rotation and exposure adjustment. The 12 gray-level gradient co-occurrence matrix texture features of the images were extracted to construct a feature vector, establishing the training dataset and test dataset. Selection of the SVM kernel function, the GA optimization parameter setting, and the base classifier number was discussed. The coal gangue image recognition effects of the AdaB-GA-SVM classifier and the other strong classifiers with different base SVM classifiers were investigated. The results indicated that the recognition accuracy of GA-SVM was the best when the kernel function of SVM was RBF and the population number, crossover probability, and mutation probability were 80, 0.9, and 0.005, respectively. The AdaB-GA-SVM classifier has excellent identification and effective classification performance with the highest accuracy of 95%, a precision rate of 92.8%, recall rate of 97.3%, and KS values of 0.79.