Temperature distribution in concrete is significant to the concrete structure’s macro properties and different factors affect the heat transfer in concrete, and therefore influence the temperature distribution. This work established a three-dimensional transient heat transfer model coupled with various environmental factors, using the finite element method for calculating the results and real-measured data for testing accuracy. In addition, a sensitivity evaluation of various factors was conducted. Due to various environmental factors, the results revealed that the prediction of temperature distribution in concrete by the three-dimensional model had great accuracy with an error of less than 4%. A particular hysteresis effect of temperature response in the concrete existed. Considering heat transfer in different spatial directions, the model can predict the temperature change of each spatial point instead of the spatial surface in different depths, proving the shortcomings of a one-dimensional heat transfer model. A greater solar radiation intensity caused a more significant temperature difference on the concrete surface: the surface temperature difference in July was twice as significant as that in December. Wind speed had a cooling effect on the concrete surface, and stronger wind speed accompanied with a stronger cooling effect made the surface temperature closer to the ambient temperature. Material properties had different effects on the temperature distribution of the surface part and internal part: the specific heat capacity determined the speed of the outer layer temperature change while the thermal conductivity determined the speed of the inner layer temperature change.