High-modulus asphalt, with its excellent fatigue resistance and high-temperature resistance, is gradually becoming a preferred material for the development of durable asphalt pavements. However, its poor low-temperature performance has become one of the key bottlenecks restricting its wide application. In recent years, in-depth analysis of the mechanism underlying the changes in the low-temperature performance of high-modulus asphalt has gradually become a research focus in the field of asphalt pavements. Accordingly, this study selected four representative high-modulus asphalts, conducted bending beam rheometer (BBR) tests to obtain their low-temperature creep parameters, and used three viscoelastic constitutive models to investigate their low-temperature constitutive relationships. Grey relational analysis (GRA) was further applied to evaluate the models. The results show that, when evaluating the low-temperature performance of high-modulus asphalt, the elastic and viscous parameters variation laws, for the three-parameter solid (TPS) model and four-parameter solid (FPS) model, are not obvious and have large fluctuations, and the accuracy of the fitting curves is relatively low, while the Burgers model has extremely high fitting accuracy, with small parameter fluctuations and significant regularity. The GRA model reveals that the Burgers model is more suitable than the TPS and FPS models for describing the low-temperature creep behavior of high-modulus asphalt, which further confirms the reliability of using the Burgers model to evaluate the low-temperature performance of high-modulus asphalt.