The evaluation and renovation of existing building envelope has important practical significance for energy conservation and emission reduction in the field of architecture. With the development of digital cities, 3D models with rich temperature information can realize the comprehensive and accurate detection and evaluation of the existing building envelope. However, the 3D model reconstructed from thermal infrared images has only relative temperature distribution and no temperature value of each location, so it is impossible to quantify the extent of the defect from it. To solve this issue, this paper develops a method to establish a 3D point cloud model with temperature information at selected points. The proposed 3D model is generated based on the thermal infrared images acquired by an unmanned aerial vehicle (UAV) equipped with an infrared camera. In the generated 3D thermal infrared model, we can not only get the relative temperature distribution of the building’s full envelope structure, but also obtain the exact temperature value of any selected point. This method has been verified by field measurements and the result shows that the deviation is within 5 °C. In addition to temperature information, the generated 3D model also has spatial and depth information, which can reflect the appearance information and 3D structure of the monitoring target more realistically. Thus, by using this method, it is possible to achieve a comprehensive, accurate, and efficient on-site assessment of the building envelope in the urban area.