The assembly quality of scaffolding systems directly governs the safety of personnel on construction sites. According to construction safety statistics, scaffolding-related accidents account for approximately 30–40% of construction fatalities globally, with geometric assembly deviations being a contributing factor in over 60% of scaffold collapse incidents. Traditional scaffolding inspections rely heavily on manual measurements, which are inherently inefficient, hazardous, and difficult to scale comprehensively. This study presents an automated evaluation framework that integrates computer vision with structural mechanics simulations. First, an object detection model based on the SegFormer encoder architecture is developed to precisely identify scaffolding standards, ledgers, and couplers against complex site backgrounds. Its hierarchical Transformer encoder and global self-attention mechanism enable the model to capture long-range topological relationships, achieving a mean Average Precision (mAP@0.5) of 95.2% on a custom dataset with an inference speed of 45 FPS per 640 × 640 image patch. For complete high-resolution frame processing including tiling and geometric extraction, the end-to-end pipeline requires approximately 8–12 s per frame. Second, a simplified Hough transform with a restricted parameter domain is introduced. Integrated with a dual-track image processing workflow, this algorithm performs sub-pixel centerline fitting to automatically extract critical geometric parameters, including lift height and bay width, maintaining a relative measurement error within 3.5% compared to manual ground truth. Finally, a parameterized finite element model is established. An automated mapping middleware dynamically injects the extracted as-built parameters into the simulation environment. Comparative simulation analysis indicates that a 14.7% deviation in standard lift height, coupled with an initial tilt defect of 1/150, precipitates a 22.4% reduction in the predicted structural stability factor, illustrating the framework’s capability for assessing relative capacity degradation between design intent and as-built conditions. This framework establishes a robust, closed-loop pipeline spanning visual perception and structural safety assessment, indicating potential for automated construction site safety management.

To address challenges such as complex background interference, limited long-range modeling capabilities of CNNs, and poor generalization in steel-concrete cross-material scenarios, this study proposes an enhanced detection framework. This framework integrates a TAPFormer with morphological reasoning rules. The method utilizes TAPFormer as the backbone network. It captures global topological features of cracks through a Task-Aware Query mechanism. This approach compensates for the deficiencies of traditional convolutional operators in modeling the continuity of thin and long cracks. Furthermore, a mask reasoning module based on geometric priors is developed to handle unstructured interferences, such as marker pen marks, welds, and concrete holes. This module defines logical criteria, including edge curvature consistency, axial aspect ratios, and endpoint extension directions. These criteria are used to perform topological repair and filter false positives in the initial segmentation masks. A hybrid dataset containing 4500 cross-material damage images was used for validation. The results show that the proposed method achieves a mean IoU of 86.72% and an F1-score of 90.36%. Notably, the method filters over 91.0% of false positives caused by manual marker pen marks in interference-rich scenarios. Compared to mainstream state-of-the-art models, the IoU improves by at least 5.48%. The results show that the proposed framework improves the robustness and logical self-consistency of crack identification in complex engineering environments.