The sleeping and eating behaviors of horses are important indicators of their health. With the development of the modern equine industry, timely monitoring and analysis of these behaviors can provide valuable data for assessing the physiological state of horses. To recognize horse behaviors in stalls, this study builds on the SlowFast algorithm, introducing a novel loss function to address data imbalance and integrating an SE attention module in the SlowFast algorithm’s slow pathway to enhance behavior recognition accuracy. Additionally, YOLOX is employed to replace the original target detection algorithm in the SlowFast network, reducing recognition time during the video analysis phase and improving detection efficiency. The improved SlowFast algorithm achieves automatic recognition of horse behaviors in stalls. The accuracy in identifying three postures—standing, sternal recumbency, and lateral recumbency—is 92.73%, 91.87%, and 92.58%, respectively. It also shows high accuracy in recognizing two behaviors—sleeping and eating—achieving 93.56% and 98.77%. The model’s best overall accuracy reaches 93.90%. Experiments show that the horse behavior recognition method based on the improved SlowFast algorithm proposed in this study is capable of accurately identifying horse behaviors in video data sequences, achieving recognition of multiple horses’ sleeping and eating behaviors. Additionally, this research provides data support for livestock managers in evaluating horse health conditions, contributing to advancements in modern intelligent horse breeding practices.