Based on mixed lubrication analysis, considering the influence of rough interface contact stiffness, contact damping, and interface friction on the gear transmission system, the relationship between interface contact and the overall performance of the gear transmission system has been established. First, the surface topography is characterized using statistical parameters of rough surfaces, and the contact stiffness and damping for tooth surfaces with different roughnesses are calculated. Subsequently, a six degree of freedom gear tribo-dynamics coupling model is developed. Finally, the established tribo-dynamics model is employed to investigate the relationship between surface roughness and the overall performance of the gear transmission system. This study provides a more intimate connection between the contact interface and the general behavior of the gear transmission system, enabling a better representation of real-world engineering problems. The research findings reveal that contact stiffness and damping decrease with increasing surface roughness. Higher roughness leads to greater gear vibration amplitude. Moreover, elevated surface roughness results in intensified meshing force and more significant energy loss. Surprisingly, when the roughness is appropriate, gears with rough surfaces lose less energy than those with smooth surfaces.