Acidic aggregates have the merits of high strength and good abrasion resistance capacity. However, its poor adhesion with asphalt binder constrains its application in pavement construction. Among these, the granite aggregate is the typical one. Therefore, this study modified granite aggregates’ surface to improve their adhesion property with the asphalt binder. Specifically, the silane coupling agent (SCA) KH-560 was adopted to achieve the modification purpose. Subsequently, asphalt mixtures with modified and unmodified granite, basalt, and limestone were subjected to the boiling test, immersion test, and freeze-thaw splitting test to estimate the asphalt adhesion property. Moreover, a molecular dynamic simulation was employed to characterize the asphalt-aggregate interface from the molecular scale. The radius distribution function (RDF) and interaction energy were used as the primary indicators. The results showed that the SCA could efficiently improve the adhesion between asphalt and granite aggregates, comparable with the alkaline aggregates. In terms of the molecular scale, the incorporation of SCA could significantly increase the concentration distribution of asphalt molecules on the aggregate surface. Meanwhile, the interaction energy was correspondingly increased due to the considerable growth of non-bond interaction.