The meshing surfaces of a gear pair are rough from a microscopic perspective and the surface topography will affect the dynamic response. To study the influence of real surface topography on the gear system dynamic performance, this paper establishes a 3‐degree of freedom transverse‐torsional dynamic model with regard to the morphology of the interface. By fractal theory, the expression of backlash between gears is modified based on the height of asperities. The time‐varying stiffness is calculated according to the fractal method rather than assuming a constant, which is more realistic. The dimensionless dynamic differential equations are established and solved with surface topography affected backlash function and time‐varying stiffness. The dynamic response of the gear system with respect to fractal dimension and fractal roughness is analyzed.