Defining failure and developing a unified failure criterion for the fatigue testing of asphalt materials remain a challenge. This study seeks to develop a failure criterion for the fatigue testing of asphalt binders under cyclic loading in the dynamic shear rheometer. Newly developed pseudo-strain energy (PSE)-based failure analysis is introduced for both the time sweep fatigue test and the accelerated linear amplitude sweep (LAS) test (AASHTO TP101). The presented methodology builds upon recent advances in the simplified viscoelastic continuum damage (S-VECD) modelling of asphalt mixtures. Trends in stored PSE have been proven to be effective in defining failure for the LAS tests of asphalt binders. This new proposed failure definition is material-dependent and, thus, is effective in capturing the benefits of asphalt modification for binder fatigue resistance. In addition, it is found that a unique relationship that is independent of loading history exists between the PSE release rate and fatigue life. The fatigue life predictions using this relationship and the S-VECD model are in reasonable agreement with the laboratory-measured fatigue life data and also generally relate well with the field fatigue performance measured in the FHWA-ALF (Federal Highway Administration – Accelerated Loading Facility) pooled fund study.