Rail fasteners are among the key components of ballasted track of high-speed railway due to their functionality of fixing rails to sleepers. The failure of rail fastening system hinders the transmission of train loads to underlying track substructure and therefore endangers the operation safety and longevity of ballasted track. This paper first established a three-dimensional (3D) numerical model of the train-ballasted track-subgrade coupling system by integrating multibody dynamics (MBD) and finite element method (FEM). Numerical simulations were then performed to investigate the effects of different patterns of rail fastener failure (i.e., consecutive single-side, alternate single-side, and consecutive double-side) on critical dynamic responses of track structures, train running stability, and operation safety. The results show that the resulting influences of different patterns of rail fastener failure descend in the order of consecutive double-side failure, consecutive single-side failure, and alternate single-side failure. As the number of failed fasteners increases, the range where dynamic responses of track structures are influenced extends, and the failure of two consecutive single-side fasteners exerts a similar influence as that of four alternate single-side fasteners. The failure of single-side fasteners affects dynamic responses of the intact side of track structures relatively insignificantly. The influence of rail fastener failure on track structures exhibits hysteresis, thus indicating that special attention needs to be paid to locations behind failed fasteners during track inspection and maintenance. The occurrence of the failure of two or more consecutive fasteners demands timely maintenance work in order to prevent aggravated deterioration of track structures. The findings of this study could provide useful reference and guidance to smart track condition assessment and condition-based track maintenance.