The transmission accuracy and meshing performance of the gearbox is determined by the internal helical gears pair. Thermal deformation of internal helical gears pair is derived from sliding friction between the contacting teeth surface, resulting in shock, vibration, and misalignments. The purpose of this paper is to compare the influence of a modified gear and an unmodified gear on the temperature field and transmission characteristics of a planetary gear system under the same working conditions. This study presents an innovative temperature field model for gear pairs utilizing Surf152 elements, integrating Hertzian contact theory, tribological principles, and finite element analysis. For the first time, we quantitatively demonstrate the enhancement of thermo-mechanical performance through topological modification in helical gears. Under light-load conditions (200 rpm), the modified gear configuration exhibits a 6.38% reduction in tooth surface temperature and a 46.5% decrease in thermal deformation compared to conventional designs. Experimental validation confirms these improvements, showing an average 62.35% reduction in transmission error. These findings establish a novel methodology for high-precision gear design while providing critical theoretical foundations for planetary gear systems, ultimately leading to significant improvements in both transmission accuracy and operational lifespan.