The condition monitoring of gearboxes is crucial to ensuring the reliability and efficiency of modern industrial machinery. The accurate estimation of Time-Varying Mesh Stiffness (TVMS) is a key aspect of modeling gear meshing behavior and generating vibration signals used for fault diagnosis. In this study, TVMS is calculated by using the Refined Finite Element Method (R-FEM), which captures detailed gear-body compliance and distributed load effects. The dynamic model of a two-stage gearbox is then used to generate vibration responses under both healthy and faulty conditions. A comprehensive parametric sensitivity analysis is conducted on critical gear modeling parameters, including tooth profile deviations, mesh convergence in contact zones, assembly tolerance-induced interaxial variations, load-dependent stiffness variations, and hub-radius effects. Experimental validation using a gearbox test bench confirms that the proposed methodology accurately reproduces fault-specific harmonic components. These results indicate that the hybrid FEM–dynamic modeling approach effectively balances accuracy and computational efficiency, thereby providing a robust framework for advanced fault detection and maintenance strategies in gear systems.