To address the issues of significant brittleness and susceptibility to catastrophic failure in traditional gangue-based cemented backfill, this study designed and prepared steel fiber and fly ash reinforced gangue backfill (FGB). Through macroscopic mechanical testing and Discrete Element Method (DEM) numerical simulation, the synergistic effects of steel fiber (VSF=0.0~2.0%) and fly ash (ωFA=5~15%) content on the mechanical properties, energy evolution, and failure modes of the FGB were systematically investigated. The results indicate that steel fibers and fly ash significantly improve the mechanical properties and failure modes of the backfill. The steel fiber content exhibits a linear positive correlation with the strength of the backfill, while the fly ash content shows a negative correlation with strength. The incorporation of fly ash enhances energy dissipation, and steel fibers effectively regulate the distribution ratio between elastic energy and dissipated energy. The addition of steel fibers and fly ash caused the crack type in the backfill to shift from a predominantly S-type to a composite S, T, and ST failure mode, and the failure mechanism transformed from a brittle mode controlled by a single dominant crack to a ductile mode characterized by multi-crack propagation. The fibers restructured the three-dimensional stress chain network, altering stress transfer pathways, increasing horizontal contact forces and the density of stress chains, and enabling the gradual transfer of stress from localized failure zones to the entire structure. This study provides a reference for the development of high-toughness, high-performance mine backfill materials.