Protective behavior in dairy cattle is one of the important potential indicators of their health and welfare status, and the precise detection of this behavior is of great significance for improving pasture management. However, existing methods face challenges, including capturing rapid motions, excessive background interference, and sample imbalance in complex agricultural environments. In response to these challenges, we proposed a Multi-Stage Attention SlowFast (MSA-SlowFast) model based on the improved SlowFast network to explore the model’s ability to distinguish between normal and protective behavior of dairy cattle. It achieves performance improvement through three core modules: the Multi-Path Balanced Head (MPBHead) for alleviating category imbalance, the Spatio-Temporal Convolutional Block Attention Module (ST-CBAM) for enhancing key feature extraction, and the 7 (BAF) for promoting multi-path feature complementarity. Additionally, we proposed novel timing-aware oversampling methods and dynamic loss adjustment mechanisms to further improve the detection performance of minority-class protective behaviors. Finally, a spatio-temporal-oriented dairy cattle protective behaviors dataset is constructed. Experimental results demonstrate that the proposed MSA-SlowFast model achieves 79.41% mAP, surpassing the standard SlowFast (70.58%) and Slow-only (68.21%). Further validation shows that the model exhibits high detection confidence in four specific actions labeled as protective behavior: 0.97 for tail swaying, 0.90 for head shaking, 0.92 for ear flapping, and 0.90 for leg kicking. These preliminary results show that the method proposed in this study has certain feasibility and reference value for the detection of protective behavior of dairy cattle under our constructed dataset.