The vibrations of a shaft in rotary mechanical systems supported by angular contact ball bearings are investigated theoretically and experimentally for various preloads in this paper. In the theoretical part of the study, a dynamic bearing model is presented, a rigid shaft supported by EHL angular contact bearing has been modelled as 5 DoF. Non-linear equations of motion are solved numerically by the Runge–Kutta method. In the second part of the study, an experimental setup that enables performing different operating cases has been designed to validate the theoretical results. Theoretical and experimental data are investigated and compared in both time and frequency domains and the results are compared. It is observed from both the theoretical and experimental studies that preload has a significant effect on the vibration behaviour. Results show that the increase in preload reduces the amplitude of the variable compliance frequencies of bearing, the natural frequency of system is shifted to a higher value, and using signal processing with an envelope spectrum gives better results in spectral analysis; small deviations occur between the theoretical and the experimental data due to modulation and noise from machine elements such as gear, motor, misalignment, waviness, etc. Therefore, the presented dynamic bearing model can be used with a reasonable accuracy to examine effect of preload on the vibration of shaft-bearing system.