Traditional structural damage detection relies on multi-sensor arrays (e.g., total stations, accelerometers, and GNSS). However, these sensors have some inherent limitations such as high cost, limited accuracy, and environmental sensitivity. Advances in computer vision technology have driven the research on vision-based structural vibration analysis and damage identification. In this study, an optimized Lucas–Kanade optical flow algorithm is proposed, and it integrates feature point trajectory analysis with an adaptive thresholding mechanism, and improves the accuracy of the measurements through an innovative error vector filtering strategy. Comprehensive experimental validation demonstrates the performance of the algorithm in a variety of test scenarios. The method tracked MTS vibrations with 97% accuracy in a laboratory environment, and the robustness of the environment was confirmed by successful noise reduction using a dedicated noise-suppression algorithm under camera-induced interference conditions. UAV field tests show that it effectively compensates for UAV-induced motion artifacts and maintains over 90% measurement accuracy in both indoor and outdoor environments. Comparative analyses show that the proposed UAV-based method has significantly improved accuracy compared to the traditional optical flow method, providing a highly robust visual monitoring solution for structural durability assessment in complex environments.