The adhesion between bitumen and mineral composition plays a vital role for the performance of asphalt mixtures. This study compares the adhesion of limestone, dolomite, and granodiorite to bitumen and evaluates the effects of different mineral components on adhesion. Three kinds of aggregates were tested through rolling-bottle tests. Afterwards, the respective fillers were integrated into asphalt mastic in a 1.6:1 mass ratio with bitumen and were subjected to frequency scan tests separately. A modified Luis Ibrarra-A model, K. Ziegel-B model, and K-B-G model were used to evaluate the bitumen–filler interactions based on the rheology of the asphalt mastic. In addition, the interface behavior between eight mineral components from these fillers/aggregates and bitumen were investigated by molecular dynamics (MD) simulations. The work of the adhesion and molecule concentration profiles were obtained from MD simulations. The results showed that the limestone and dolomite had better interfacial adhesion to the bitumen than the granodiorite. The calcium oxide and titanium oxide had the highest potential adsorption effect on the bitumen. Moreover, the high calcium oxide content contributed to better bitumen adhesion with the limestone and dolomite than with the granodiorite, which was further confirmed by additional molecule concentration profile analysis. This research contributes to the in-depth understanding of the effect of different chemical properties on the performance of asphalt mastic and the selection of suitable mineral components to improve the bitumen–filler/aggregate interface and asphalt mixture performance in general.