Fish object detection has attracted significant attention because of the considerable role that fish play in human society and ecosystems and the necessity to gather more comprehensive fish data through underwater videos or images. However, fish detection has always faced difficulties with the occlusion problem because of dense populations and underwater plants that obscure them, and no perfect solution has been found until now. To address the occlusion issue in fish detection, the following effort was made: creating a dataset of occluded fishes, integrating the innovative modules in Real-time Detection Transformer (RT-DETR) into You Only Look Once v8 (YOLOv8), and applying repulsion loss. The results show that in the occlusion dataset, the mAP of the original YOLOv8 is 0.912, while the mAP of our modified YOLOv8 is 0.971. In addition, our modified YOLOv8 also has better performance than the original YOLOv8 in terms of loss curves, F1–Confidence curves, P–R curves, the mAP curve and the actual detection effects. All these indicate that our modified YOLOv8 is suitable for fish detection in occlusion scenes.