In this study, a nanocrystalline layer composed primarily of martensite phase was prepared on the surface of 304 stainless steel. Furthermore, the martensite phase content in the nanocrystalline layer was adjusted by heat treatment at 500 °C and 550 °C, respectively, and the cavitation erosion resistance of the nanocrystalline layer before and after heat treatment was investigated. The results showed that the nanocrystalline layer before and after heat treatment exhibited excellent erosion resistance, with cumulative mass loss of approximately 1/7, 1/5, and 1/3 that of the traditional 304 stainless steel, respectively. The nanocrystalline layer could significantly inhibit the growth of cavitation pits due to the high density of grain boundaries. However, due to the decrease in hardness of the nanocrystalline layer after heat treatment, the propagation speed of cavitation cracks was accelerated, and the cavitation erosion performance of the nanocrystalline layer showed a downward trend.