Semi-flexible pavement (SFP) is widely used in recent years because of its good rutting resistance, but it is easy to crack under traffic loads. A large number of studies are aimed at improving its crack resistance. However, the understanding of its fatigue resistance and fatigue-cracking mechanism is limited. Therefore, the semi-circular bending (SCB) fatigue test is used to evaluate the fatigue resistance of the SFP mixture. SCB fatigue tests under different temperature values and stress ratio were used to characterize the fatigue life of the SFP mixture, and its laboratory fatigue prediction model was established. The distribution of various phases of the SFP mixture in the fracture surface was analyzed by digital image processing technology, and its fatigue cracking mechanism was analyzed. The results show that the SFP mixture has better fatigue resistance under low temperature and low stress ratio, while its fatigue resistance under other environmental and load conditions is worse than that of asphalt mixture. The main reason for the poor fatigue resistance of the SFP mixture is the poor deformation capacity and low strength of grouting materials. Furthermore, the performance difference between grouting material and the asphalt binder is large, which leads to the difference of fatigue cracking mechanism of the SFP mixture under different conditions. Under the fatigue load, the weak position of the SFP mixture at a low temperature is asphalt binder and its interface with other materials, while at medium and high temperatures, the weak position of the SFP mixture is inside the grouting material. The research provides a basis for the calculation of the service life of the SFP structure, provides a reference for the improvement direction of the SFP mixture composition and internal structure.