The historical Lingnan gray brick buildings in Macau, a World Heritage Site, are facing severe deterioration due to prolonged disrepair, manifesting as cracks, breakages, moss adhesion, and other types of surface damage. These issues threaten not only the structural stability of the buildings but also the conservation of cultural heritage. To address the inefficiencies and low accuracy of traditional manual inspections, this study proposes an automated recognition and quantitative detection method for wall surface damage based on the YOLOv8 deep learning object detection model. A dataset comprising 375 annotated images collected from 162 gray brick historical buildings in Macau was constructed, covering eight damage categories: crack, damage, missing, vandalism, moss, stain, plant, and intact. The model was trained and validated using a stratified sampling approach to maintain a balanced class distribution, and its performance was comprehensively evaluated through metrics such as the mean average precision (mAP), F1 score, and confusion matrices. The results indicate that the best-performing model (Model 3 at the 297th epoch) achieved a mAP of 61.51% and an F1 score up to 0.74 on the test set, with superior detection accuracy and stability. Heatmap analysis demonstrated the model’s ability to accurately focus on damaged regions in close-range images, while damage quantification tests showed high consistency with manual assessments, confirming the model’s practical viability. Furthermore, a portable, integrated device embedding the trained YOLOv8 model was developed and successfully deployed in real-world scenarios, enabling real-time damage detection and reporting. This study highlights the potential of deep learning technology for enhancing the efficiency and reliability of architectural heritage protection and provides a foundation for future research involving larger datasets and more refined classification strategies.