As the key components of sealing applications, rubber seals are subject to complicated environmental conditions during the service lifetime. In this study, the aging of three typical rubber materials, ethylene–propylene–diene monomer rubber, liquid silicone rubber, and fluorine rubber, was tested under different high- and low-temperature cycle aging environments. The experimental results confirm that the reciprocating temperature cycle causes a type of fatigue failure, which could result in an increase in the rubber compression set. In addition, a novel accelerated aging test method was proposed based on the dominant damage mechanism of rubber material caused by the temperature cycle treatments. Based on this method, the long-term aging test results of rubber samples under high- and low-temperature cycle conditions can be predicted. This method could significantly shorten the aging test time and reduce the test cost.