Road construction consumes great amounts of high-grade natural resources. Using low-grade natural rocks or some solid wastes as substitute materials is a hot topic. Considering this, the feasibility of using low-grade granite aggregate, solid waste-based filler (desulphurization gypsum residues, DGR) and binder (waste tire rubber modified asphalt, RMA) simultaneously in asphalt mixtures has been fully investigated in this research. The commonly used base asphalt and limestone powder (LP) filler were control groups. Material characteristics of raw materials mainly including micro-morphology, functional group, mineral phase, chemical composition and thermal stability were first evaluated in order to recognize them. Four asphalt mixtures (two asphalt binder and two filler) were then designed by standard Superpave method. Finally, a detailed investigation into the pavement performance of asphalt mixtures was carried out. The moisture damage resistance and low-temperature crack resistance were detected by the changing rules of stability, strength and fracture energy, and the high-temperature stability and fatigue performance were determined by wheel tracking test and indirect tensile (IDT) fatigue test, respectively. Results suggested that RMA and DGR both showed positive effects on the low-temperature crack resistance and fatigue property of the granite asphalt mixture. DGR also strengthened moisture stability. The contribution of RMA on high-temperature deformation resistance of the granite asphalt mixture was compelling. It can offset the insufficiency in high-temperature stability made by DGR. A conclusion can be made that asphalt mixture prepared with granite, DGR and RMA possesses satisfactory pavement performances.