In this paper, ordinary Portland cement, ultrafine cement, polyurethane, and epoxy resin were selected as typical grouting materials. Grouting simulation tests were first conducted to prepare the grouted concrete crack sample. The effect of concrete crack parameters (i.e., crack aperture and roughness), grout water–cement ratio, and grouting pressure on the water-plugging performance of different grouting materials was explored through the impermeability test. The microstructure of grouted concrete cracks was analyzed by means of scanning electron microscopy (SEM) and computed tomography (CT), and the difference in water-plugging performance of different grouting materials was explained at the micro level. The results show that the impermeability of the four grouting materials was ranked as follows: Epoxy resin > polyurethane > ultra-fine cement > ordinary Portland cement. The concrete cracks grouted by epoxy resin have the highest plugging failure water pressure and the lowest permeability, which is the optimal grouting material. The effectiveness of crack grouting in water-plugging was directly proportional to the grouting pressure, provided the pressure did not exceed a certain value. When the pressure surpassed the threshold, the increase in pressure did not have a significant impact on the water plugging performance. For the two cement-based materials, the threshold pressure was 1 MPa, while for the other two chemical grouts, it was 2 MPa. The two cement-based grouts with a water–cement ratio of 0.8 showed optimal water-plugging performance. The water-plugging performance of ordinary Portland cement paste, ultra-fine cement pastes, and polyurethane grout was negatively correlated with crack aperture and positively correlated with crack roughness. However, the water-plugging performance of epoxy resin grout was not affected by crack aperture or roughness.