Fish feeding behavior analysis plays a critical role in aquaculture, marine fisheries, and ecological monitoring. At present, the accuracy of current fish feeding behavior analysis is easily affected by the environment, and the hypoxia state in water and high temperature have a huge impact on behavior. In addition, the poor stability, and the speed is affected by the calculation amount of the model, so it cannot be applied to aquaculture ponds. In order to solve the above problems, we propose an improved RepVGG-based method for fish feeding behavior analysis to identify nine behavioral states of fish populations, including hypoxia, hot, and normal states of none, medium, and strong feeding. Firstly, our method adds the identity and residual branches to the block of the VGG network, focusing on the acceleration operation to ensure accuracy. At the same time, the Efficient Channel Attention (ECA) module is added to reduce dimensionality to balance speed and accuracy. To evaluate the effectiveness of the method, it is validated on a constructed fish feeding behavior dataset and compared with a Convolutional Neural Network (CNN) including AlexNet, VGG16, ResNet34, MobileNet V3, and RepVGG. The experimental results show that after 15 epochs of training, the average precision of proposed method reached 97%. Compared with the basic classification algorithm, our method can increase by at least 3% while the inference speed exceeds 85 Frames Per Second (FPS). This study can be integrated into the aquaculture vision system to guide users to plan feeding strategies, and provide new ideas and methods for water quality monitoring.