Infrared cameras form images by capturing the thermal radiation emitted by objects in the infrared spectrum, making them complex sensors widely used in maritime surveillance. However, the broad spectral range of the infrared band makes it susceptible to environmental interference, which can reduce the contrast between the target and the background. As a result, detecting infrared targets in complex marine environments remains challenging. This paper presents a novel and enhanced detection model developed from the real-time detection transformer (RT-DETR), which is designated as MAFF-DETR. The model incorporates a novel backbone by integrating CSP and parallelized patch-aware attention to enhance sensitivity to infrared imagery. Additionally, a channel attention module is employed during feature selection, leveraging high-level features to filter low-level information and enabling efficient multi-level fusion. The model’s target detection performance on resource-constrained devices is further enhanced by incorporating advanced techniques such as group convolution and ShuffleNetV2. The experimental results show that, although the enhanced RT-DETR algorithm still experiences missed detections under severe object occlusion, it has significantly improved overall performance, including a 1.7% increase in mAP, a reduction in 4.3 M parameters, and a 5.8 GFLOPs decrease in computational complexity. It can be widely applied to tasks such as coastline monitoring and maritime search and rescue.