The penetration grade system is still widely adopted for selecting asphalt binder with desired paving performance. However, the initial material compositions of asphalt binder with the same penetration level are still different, and vary with the crude oil source and essentially result in different rheological performance. This study aimed to assess the linear viscoelastic (LVE) properties, and high- and intermediate-temperature and microscale characteristics of seven unmodified asphalt binders from different sources and countries with the same penetration level of 70. The LVE parameters were firstly evaluated followed by comparisons to various damage-based indexes. The microstructure of asphalt binders was further investigated followed by correlations between morphology and performance parameters. Experimental results indicate the |G*|/sin δ is well related to the MSCR-based non-recoverable creep compliance; furthermore, the R and |G*|·sin δ can generally represent the LAS-based failure strain and fatigue life, respectively. The viscoelastic nature of tested binders was clearly distinguished and related to rheological performance by atomic force microscopy (AFM). The roughness parameters and the phases’ content derived from AFM images showed significant correlations with LVE characteristics and fatigue resistance nature, respectively. This research provides theoretical foundations for further investigating the rheological performance and microstructure characteristics, and their correlations with asphalt binders.