Infrared thermography (IRT), a widely used nondestructive testing method, is commonly employed to identify moisture in historic walls. However, its reliance on manual interpretation by experts makes the process both time-consuming and costly. This study addresses the challenge of detecting wall moisture; this issue is closely linked to the deterioration of cultural heritage structures. This study focuses on the brick walls of the Tainan Confucian Temple, the oldest Confucian temple in Taiwan. The targeted are walls coated with lime plaster mixed with red mineral pigments, a traditional finish that gives the temple its distinctive red appearance. This study proposes a system to automatically identify wall areas, mark low-temperature zones, and determine the presence and distribution of moisture. Visible and infrared thermal images of these walls are captured and preprocessed to normalize the size and enhance the features. Finally, two convolutional neural network (CNN) models are trained in this study: one for identifying wall regions and the other for detecting low-temperature areas. The proposed method achieves an accuracy of 91.18% in detecting wall moisture, representing a 24.05% improvement over conventional object recognition techniques, the accuracy of which is 73.5%. In addition, this method requires only 3 s to detect the wall moisture, representing a 99.92% reduction in processing time compared to the conventional manual method. This method not only provides a fast and objective method for assessing moisture in lime-plastered heritage walls but also significantly enhances the efficiency of restoration efforts. This method can be applied to similar wall structures in other Confucian temples, offering broad potential for heritage conservation.