In this work, a thermal degradation mechanism of room temperature vulcanized (RTV) phenyl silicone rubber that was vulcanized by different crosslinking agents was discussed. Firstly, RTV phenyl silicone rubber samples were prepared by curing hydroxyl-terminated polymethyldiphenylsiloxane via three crosslinking agents, namely, tetraethoxysilane (TEOS), tetrapropoxysilane (TPOS), and polysilazane. Secondly, the ablation properties of RTV phenyl silicone rubber were studied by the muffle roaster test and FT-IR. Thirdly, thermal stability of the three samples was studied by thermogravimetric (TG) analysis. Finally, to explore the thermal degradation mechanism, the RTV phenyl silicone rubber vulcanized by different crosslinking agents were characterized by TG analysis-mass spectrum (TG-MS) and pyrolysis gas chromatogram-mass spectrum (pyGC-MS). Results showed that the thermal stability of RTV phenyl silicone rubber is related to the amount of residual Si–OH groups. The residual Si–OH groups initiated the polysiloxane chain degradation via an ‘unzipping’ mechanism.