This study presents a modular diagnostic framework for evaluating thermal degradation in aging building envelopes by integrating infrared thermography, panoramic reconstruction, and deep learning-based semantic segmentation into a unified workflow. The methodology combines image registration, panoramic synthesis, façade component segmentation, and quantitative surface temperature analysis to provide scalable and reproducible diagnostics. By excluding fenestration zones—where infrared measurements are physically unreliable—the framework focuses on opaque wall regions and window surroundings to ensure physically meaningful evaluation. Field validation was conducted on a multi-story office building constructed in 1996. The diagnostic indicators revealed a mean wall surface temperature of 14.3 °C with a standard deviation of 5.6 °C, and a temperature factor ranging from 0.67 to 0.78 under measured conditions. The vulnerable area ratio reached 9.1% for walls, while window areas showed greater vulnerability at 12.74%, with anomalies concentrated at frame–glass interfaces and perimeter seals. These quantitative results confirmed the framework’s ability to detect thermal irregularities and visualize localized anomalies. More importantly, the contribution of this study lies in establishing a systematic and extensible diagnostic pipeline that advances building envelope analysis, supporting large-scale energy audits, retrofit prioritization, and sustainable building management.