Evaluation of the damage and failure sequence of key components of track-bridge system for high-speed railway under seismic action is the basis for the design of a high-speed railway system and function recovery after earthquake. However, this evaluation has not been studied well. In this study, a simply supported beam with CRTS II track system and CRH2 high-speed train were taken as research objects. A train-bridge coupled finite element model was established, and experimental verification was carried out. Ten seismic waves exhibiting the highest matching degree with the site characteristics of bridge site were selected, and a nonlinear time-history analysis was carried out. The average peak ground acceleration (PGA) was obtained when each component suffered damage and failure for the first time, and the damage and failure sequences of components were arranged to analyze the seismic damage mode and failure mechanism of track-bridge system for high-speed railway. Finally, the corresponding damage and failure levels for track-bridge system were established. According to the research findings, under longitudinal earthquake, the damage and failure sequence of components is in the order of damage and failure of abutment sliding layer, damage of fixed abutment bearing, damage and failure of shear slot, failure of fixed abutment bearing, damage and failure of fixed pier bearing, and damage and failure of pier. Under transverse earthquake, the damage failure sequence of components is in the order of damage of abutment sliding layer, damage and failure of fixed pier bearing, damage of abutment sliding layer, damage of abutment fixed bearing, damage and failure of block, failure of fixed abutment bearing, and damage and failure of shear slot and pier. The fastener, mortar layer, and shear rebar remain in an intact state under transverse and longitudinal seismic motions. The dynamic effect of trains has no significant effect on the damage mode of track-bridge system for high-speed railway and yield order of key components.