Asphalt–filler interaction plays a key role in the performances of asphalt mastic and asphalt mixture. However, the current studies ignore the filler critical volume fraction that has significant effects on asphalt–filler interaction ability. There is also lacking quantitative comparison of evaluation indices within filler critical volume fraction. In this study, two kinds of virgin asphalt binders and three kinds of mineral fillers were selected to fabricate asphalt mastics, and the dynamic shear rheological properties of virgin asphalt binders and asphalt mastics were measured, and the interaction evaluation indices complex viscosity coefficient Δη*, intrinsic viscosity [η], complex modulus coefficient ΔG*, K-B-G*, Einstein coefficient K_E, L-A-δ and K-B-δ were quantitatively compared. The results indicate that complex viscosity coefficient Δη* and intrinsic viscosity [η] are not suitable for evaluation of asphalt–filler interaction ability. ΔG* and K-B-G* can evaluate the differences of asphalt and different fillers interaction ability based on the same kind of asphalt binder. Nevertheless, ΔG* and K-B-G* cannot differentiate the effects of asphalt binder on asphalt–filler interaction ability well. Although Einstein coefficient K_E can evaluate asphalt–filler interaction ability, the Einstein coefficient K_E is fitting results, and cannot distinguish the interaction ability under different filler volume fractions. L-A-δ and K-B-δ can differentiate the interaction ability of different asphalt binders and fillers well, and also the sensitivity factors of index K-B-δ are greater than L-A-δ. Therefore, K-B-δ is the most suitable for evaluating asphalt and filler interaction ability. Based on the evaluation index K-B-δ, asphalt–filler interaction ability increases initially and decreases afterwards with an increase in filler concentration within filler critical volume fraction.