Algae are vital to aquatic ecosystems, with their structure and abundance influencing ecological health. However, automated detection in real water samples is hindered by complex backgrounds, species diversity, and size variations. Traditional methods are deemed costly and species-specific, leading to deep learning adoption. Current studies rely on CNN-based models and limited datasets. To improve the detection accuracy of multiple algal species in real, complex backgrounds, this study collected multi-species algae samples from actual water environments and implemented an integrated Transformer-based framework for automated localization and recognition of small, medium, and large algae species. Specifically, algae samples from five different regions were collected to construct a comprehensive dataset containing 25 algal species with diverse backgrounds and rich category diversity. To address dataset imbalances in minority species, a segmentation-fusion data augmentation method was proposed, which enhanced performance across YOLO, Faster R-CNN, and Deformable DETR models, with YOLO achieving a 7.1% precision increase and a 1.5% mAP improvement. Model optimization focused on an improved Deformable DETR, incorporating multi-scale feature extraction, deformable attention mechanisms, and the normalized Wasserstein distance loss function. This improvement enhanced small target and overlapping object detection, achieving a 10.4% mAP increase at an intersection over union (IoU) threshold of 0.5 and outperforming unmodified Deformable DETR.