Land surface temperature (LST) variations are very complex in mountainous areas owing to highly heterogeneous terrain and varied environment, which complicates the surface urban heat island (SUHI) in mountain cities. Previous studies on the urban heat island (UHI) effect mostly focus on the flat terrain areas; there are few studies on the UHI effect in mountainous areas, especially on the influence of elevation on the SUHI effect. To determine the SUHI in the Qinling–Daba mountains (China), MODIS LST data were first preprocessed and converted to the same elevations (1500 m, 2000 m, 2500 m, 3000 m, and 3500 m) using a digital elevation model and the random forest method. Then, the average LSTs in urban land, rural land, and cultivated land were calculated separately based on the ranges of the invariable urban, rural, and cultivated areas during 2010–2018, and the urban, rural, and cultivated land LST difference were estimated for the same elevations. Results showed that the accuracy of LST estimated using the random forest method is very high (R² ≥ 0.9) at elevations of 1500 m, 2000 m, 2500 m, 3000 m and 3500 m. The difference in urban, rural, and cultivated lands’ LST has a trend of decrease with increasing elevation, meaning that the SUHI weakens at higher elevations. The average LST of urban areas is 0.52–0.59 °C (0.42–0.57 °C) higher than that of rural and cultivated areas at an elevation of 1500 m (2000 m). The average LST of urban areas is 0.10–1.25 °C lower than that of rural and cultivated areas at elevations of 2500 m, 3000 m, and 3500 m, indicating absence of the SUHI at those elevations.