To investigate the effect of styrene-butadiene-styrene (SBS) modifier content on the viscoelastic behavior of SBS-modified asphalt (SBSMA) at different temperatures and phase structures, the star SBS modifier was chosen to fabricate seven types of SBSMA with different contents. Multiple stress creep recovery (MSCR), linear amplitude sweep (LAS), and low-temperature frequency sweep tests were adopted to study the influence of SBS modifier content on the viscoelastic performance of SBSMA at high to low temperatures. The SBSMA’s microstructure with different contents was investigated using a fluorescence microscope. The results indicated that the change in non-recoverable creep compliance and creep recovery rate was bounded by 4.5% content at high temperatures, with an apparent turning point. The changing slope of content at less than 4.5% was much higher than that of the content greater than 4.5%. At medium temperatures, the fatigue life of SBSMA increased exponentially with the rising modifier content. The rate of increase in fatigue life was the largest as the content increased from 4.5% to 6.0%. At low temperatures, the low-temperature viscoelastic property index G (60 s) of SBSMA decreased logarithmically as the modifier content increased. In terms of the microscopic phase structure, the SBS modifier gradually changed from the dispersed to the continuous phase state with the increasing SBS modifier content.