Surface roughness is one of the main factors that affect the ultrasonic testing of micro-cracks. This article theoretically analyzes the relationship between the changes in the energy intensity of crack echo waves and roughness-modified transmission coefficients. A series of simulations are carried out using two-dimensional sinusoidal curves as rough surface. Then, parallel experiments are performed on sample surfaces with different arithmetic average heights (R_a). The signal amplitude ratio factor (SARF) is defined to assess the ultrasonic detection capacity for micro-cracks. Both finite element analysis and experimental results show that signal amplitude decreases with an increase in R_a, resulting in signal-to-noise ratio loss. Amplitude attenuation caused by the rough back surface is less than that caused by the rough front surface. It is difficult to identify the signal of micro-cracks with a depth less than 400 μm when the R_a of the front surface is larger than 15 μm. Cracks with depths of more than 200 μm can be distinguished when the back-surface roughness is less than 24 μm. Furthermore, the amplitude of the micro-crack signal increases slightly with variation in the horizontal parameter (R_sm). This study provides a valuable reference for the precision evaluation of micro-cracks using ultrasonic inspection.