Phase-sensitive optical time domain reflectometry (Φ-OTDR) is a highly sensitive distributed vibration sensing technology crucial for pipeline safety monitoring. However, its sensitivity makes it susceptible to environmental interference, leading to frequent false alarms by misclassifying routine activities as threats. To enable accurate threat identification and rapid response, this study proposes a lightweight LightPatch Vision Transformer (LP-ViT) model suitable for edge deployment. We establish a mapping between excavator-pipeline distance and threat levels: “direct intrusion” (within 5 m), “high-risk operation” (within 10 m), and “background construction” (beyond 15 m). The LP-ViT model is developed through structural optimization and parameter compression of the standard Vision Transformer, achieving a 96.6% reduction in parameter count while maintaining a high classification accuracy of 89.9%. Furthermore, via knowledge distillation, we derive an ultra-lightweight student model with merely 0.37 M parameters, which achieves an inference latency of 5.5 ms per sample, enabling millisecond-level threat detection and response. The proposed solution effectively enhances both the classification accuracy and real-time performance of Φ-OTDR systems in complex environments, providing a practical pathway for implementing edge intelligence in pipeline safety monitoring.