Accurately detecting and identifying granary pests is important in effectively controlling damage to a granary, ensuring food security scientifically and efficiently. In this paper, multi-scale images of seven common granary pests were collected. The dataset had 5231 images acquired with DSLR-shot, microscope, cell phone and online crawler. Each image contains different species of granary pests in a different background. In this paper, we designed a multi-scale granary pest recognition model, using the YOLOv5 (You Look Only Once version 5) object detection algorithm incorporating bidirectional feature pyramid network (BiFPN) with distance intersection over union, non-maximum suppression (DIOU_NMS) and efficient channel attention (ECA) modules. In addition, we compared the performance of the different models established with Efficientdet, Faster rcnn, Retinanet, SSD, YOLOx, YOLOv3, YOLOv4 and YOLOv5s, and we designed improved YOLOv5s on this dataset. The results show that the average accuracy of the model we designed for seven common pests reached 98.2%, which is the most accurate model among those identified in this paper. For further detecting the robustness of the proposed model, ablation analysis was conducted. Furthermore, the results show that the average accuracy of models established using the YOLOv5s network model combined with the attention mechanism was 96.9%. When replacing the model of PANet with BiFPN, the average accuracy reached 97.2%. At the same time, feature visualization was analyzed. The results show that the proposed model is good for capturing features of pests. The results of the model have good practical significance for the recognition of multi-scale granary pests.