To improve the accuracy, generality and convergence of thermal error compensation model based on traditional neural networks, a genetic algorithm was proposed to optimize the number of the nodes in the hidden layer, the weights and the thresholds of the traditional neural network by considering the shortcomings of the traditional neural networks which converged slowly and was easy to fall into local minima. Subsequently, the grey cluster grouping and statistical correlation analysis were proposed to group temperature variables and select thermal sensitive points. Then, the thermal error models of the high-speed spindle system were proposed based on the back propagation and genetic algorithm–back propagation neural networks with practical thermal error sample data. Moreover, thermal error compensation equations of three directions and compensation strategy were presented, considering thermal elongation and radial tilt angles. Finally, the real-time thermal error compensation was implemented on the jig borer’s high-speed spindle system. The results showed that genetic algorithm–back propagation models showed its effectiveness in quickly solving the global minimum searching problem with perfect convergence and robustness under different working conditions. In addition, the spindle thermal error compensation method based on the genetic algorithm–back propagation neural network can improve the jig borer’s machining accuracy effectively. The results of thermal error compensation showed that the axial accuracy was improved by 85% after error compensation, and the axial maximum error decreased from 39 to 3.6 µm. Moreover, the X/Y-direction accuracy can reach up to 82% and 85%, respectively, which demonstrated the effectiveness of the proposed methodology of measuring, modeling and compensating.