To verify the feasibility of applying dry flue gas desulfurization ash (DFGDA) to asphalt pavement materials, the asphalt mastic (filler and asphalt composition) prepared by adding different proportions of DFGDA and LSP (limestone powder) into 70# matrix asphalt was studied experimentally. The asphalt mastics were subjected to the penetration test, the softening point test, and the ductility test. Moreover, the rheological properties of asphalt mastic were evaluated with dynamic shear rheometer (DSR) tests and bending beam rheometer (BBR) tests. An interaction ability index C-value based on the Palierne model was proposed to evaluate the interaction ability between DFGDA and asphalt. The influence of DFGDA asphalt on the interaction ability of matrix asphalt was observed and evaluated using Fourier transform infrared spectroscopy (FTIR) and scanning electron microscopy (SEM). The results showed that with the increasing proportion of DFGDA, the penetration of asphalt mastic gradually decreased, the softening point increased, and the ductility slightly decreased. At the same temperature, the dynamic shear modulus G* of the asphalt mortar significantly increased with increasing DFGDA content. The incorporation of DFGDA negatively affected the low-temperature plastic deformation resistance of asphalt, but the impact was weakened with the growing DFGDA amount and the powder mastic ratio. The combination mode of DFGDA and matrix asphalt depends on the physical blending, and their interaction ability mainly depends on the miscibility between DFGDA and matrix asphalt. In conclusion, DFGDA can be utilized as a novel filler in asphalt pavement materials.