Accurate recognition of the flowering stage is a prerequisite for flower yield estimation. In order to improve the recognition accuracy based on the complex image background, such as flowers partially covered by leaves and flowers with insignificant differences in various fluorescence, this paper proposed an improved CR-YOLOv5s to recognize flower buds and blooms for chrysanthemums by emphasizing feature representation through an attention mechanism. The coordinate attention mechanism module has been introduced to the backbone of the YOLOv5s so that the network can pay more attention to chrysanthemum flowers, thereby improving detection accuracy and robustness. Specifically, we replaced the convolution blocks in the backbone network of YOLOv5s with the convolution blocks from the RepVGG block structure to improve the feature representation ability of YOLOv5s through a multi-branch structure, further improving the accuracy and robustness of detection. The results showed that the average accuracy of the improved CR-YOLOv5s was as high as 93.9%, which is 4.5% better than that of normal YOLOv5s. This research provides the basis for the automatic picking and grading of flowers, as well as a decision-making basis for estimating flower yield.