Pneumonia is a respiratory infection that affects the lungs. The symptoms of viral and bacterial pneumonia are similar. In order to improve automatic detection efficiency regarding X-ray images of pneumonia, this paper, we propose a novel pneumonia detection method based on the Fast-YOLO network model. First, we re-annotated the open-source dataset of MIMIC Chest X-ray pneumonia, enhancing the model’s adaptability to complex scenes by incorporating Mixup, Mosaic, and Copy–Paste augmentation methods. Additionally, CutMix and Random Erasing were introduced to increase data diversity. Next, we developed a lightweight FASPA Fast Pyramid Attention Mechanism and designed the Fast-YOLO network based on this mechanism to effectively address the complex features in pneumonia X-ray images, such as low contrast and an uneven distribution of local lesions. The Fast-YOLO network improves upon the YOLOv11 architecture by replacing the C3k2 module with the FASPA attention mechanism, significantly reducing the network’s parameter count while maintaining detection performance. Furthermore, the Fast-YOLO network enhances feature extraction capabilities when handling scenes with geometric deformations, multi-scale features, and dynamic changes. It expands the receptive field, thereby balancing computational efficiency and accuracy. Finally, the experimental results demonstrate that the Fast-YOLO network, compared to traditional convolutional neural network methods, can effectively identify pneumonia regions and localize lesions in pneumonia X-ray image detection tasks, achieving significant improvements in FPS, precision, recall, mAP @0.5, and mAP @0.5:0.95. This confirms that Fast-YOLO strikes a balance between computational efficiency and accuracy. The network’s excellent generalization capability across different datasets has been validated, showing the potential to accelerate the pneumonia diagnostic process for clinicians and enhance diagnostic accuracy.