The strapdown inertial navigation system can provide the navigation information for the boom-type roadheader in the unmanned roadway tunneling working face of the coal mine. However, the complex vibration caused by the cutting process of the boom-type roadheader may result in significant errors of its attitude and position measured by the strapdown inertial navigation system. Thus, an error compensation method based on the vibration characteristics of the roadheader is proposed in this paper. In order to further analyze the angular and linear vibration of the fuselage, as the main vibration sources of the roadheader, the dynamic model of the roadheader is formulated based on the cutting load. Following that, multiple sub-samples compensation algorithms for the coning and sculling errors are constructed. Simulation experiments were carried out under different subsample compensation algorithms, different coal and rock characteristics, and different types of roadheader. The experimental results show that the proposed error compensation algorithm can eliminate the effect of the angular and linear vibration on the measurement accuracy. The coning and sculling error of the strapdown inertial navigation system can reduce at least 52.21% and 42.89%, respectively. Finally, a strapdown inertial navigation error compensation accuracy experiment system is built, and the validity and superiority of the method proposed in this paper are verified through calculation and analysis of the data collected on the actual tunneling work face.