Livestock behavior serves as a crucial indicator of physiological health. Leveraging deep learning techniques to automatically recognize dairy goat behaviors, particularly abnormal ones, enables early detection of potential health and environmental issues. To address the challenges of recognizing small-target behaviors in complex environments, a multi-scale and lightweight behavior recognition model for dairy goats called GSCW-YOLO was proposed. The model integrates Gaussian Context Transformation (GCT) and the Content-Aware Reassembly of Features (CARAFE) upsampling operator, enhancing the YOLOv8n framework’s attention to behavioral features, reducing interferences from complex backgrounds, and improving the ability to distinguish subtle behavior differences. Additionally, GSCW-YOLO incorporates a small-target detection layer and optimizes the Wise-IoU loss function, increasing its effectiveness in detecting distant small-target behaviors and transient abnormal behaviors in surveillance videos. Data for this study were collected via video surveillance under varying lighting conditions and evaluated on a self-constructed dataset comprising 9213 images. Experimental results demonstrated that the GSCW-YOLO model achieved a precision of 93.5%, a recall of 94.1%, and a mean Average Precision (mAP) of 97.5%, representing improvements of 3, 3.1, and 2 percentage points, respectively, compared to the YOLOv8n model. Furthermore, GSCW-YOLO is highly efficient, with a model size of just 5.9 MB and a frame per second (FPS) of 175. It outperforms popular models such as CenterNet, EfficientDet, and other YOLO-series networks, providing significant technical support for the intelligent management and welfare-focused breeding of dairy goats, thus advancing the modernization of the dairy goat industry.