In this paper, phase coherence imaging is proposed to improve spatial resolution and signal-to-noise ratio (SNR) of near-surface defects in rails using cross-correlation of ultrasonic diffuse fields. The direct signals acquired by the phased array are often obscured by nonlinear effects. Thus, the output image processed by conventional post-processing algorithms, like total focus method (TFM), has a blind zone close to the array. To overcome this problem, the diffuse fields, which contain spatial phase correlations, are applied to recover Green’s function. In addition, with the purpose of improving image quality, the Green’s function is further weighted by a special coherent factor, sign coherence factor (SCF), for grating and side lobes suppression. Experiments are conducted on two rails and data acquisition is completed by a commercial 32-element phased array. The quantitative performance comparison of TFM and SCF images is implemented in terms of the array performance indicator (API) and SNR. The results show that the API of SCF is significantly lower than that of TFM. As for SNR, SCF achieved a better SNR than that of TFM. The study in this paper provides an experimental reference for detecting near-surface defects in the rails.