Polyurethane grouting technology is widely employed to maintain critical transportation infrastructure, including pavements, airports, and railways. After injection, foamed polyurethane bonds with surrounding aggregates to form a polyurethane–aggregate composite material (PACM). The gradation of aggregates in PACM, stress levels, and loading frequencies significantly influence fatigue performance under cyclic traffic loading. This study investigates the fatigue behavior of three distinct PACM gradation types through three-point bending fatigue tests under varying stress levels and loading frequencies. Results reveal that the finer gradations of PACM tend to exhibit higher flexural stiffness and longer fatigue life but also greater sensitivity to stress levels. Conversely, coarser gradations show lower stiffness but improved energy dissipation characteristics. Additionally, the flexural stiffness modulus, fatigue life, and cumulative dissipated energy decrease with increasing stress levels, while they grow with higher loading frequencies. In contrast, the dissipated angle follows an opposite trend. Additionally, mathematical models were developed to describe the evolution of dissipated energy, uncovering a three-stage pattern dominated by a prolonged plateau phase accounting for over 80% of the fatigue process. Based on this characteristic plateau, fatigue life prediction models were established for each gradation type, achieving high prediction accuracy with relative errors below 10%. These findings not only highlight the significant impact of aggregate gradation on PACM fatigue performance but also provide practical tools for optimizing material design in pavement maintenance.