Accurate high-throughput fruit detection is the core prerequisite for precision citrus management. Existing models face a critical trade-off between accuracy for small fruits and computational efficiency, restricting large-scale industry transformation. To resolve this, we propose Citrus-SGYOLOv2, an optimized deep learning architecture specifically engineered for high-throughput phenotypic monitoring. The primary contribution of this work lies in three synergistic innovations: a novel Symmetric Ghost Backbone that prunes architectural redundancy while maintaining hierarchical feature depth; a Citrus Color Prior Calibration Attention Mechanism (Citrus_SE) that embeds physiological chromaticity priors to suppress complex spectral noise from foliage; and a P2-layer-based full-scale fusion strategy designed to recover fine-grained spatial details lost during downsampling. Experiments on our self-built dataset show that Citrus-SGYOLOv2 achieves 95.54% mAP@50 and 77.13% mAP@50–95, outperforming YOLOv11s by 5.03 and 9.90 percentage points respectively. Notably, the model achieves a 48.8% reduction in parameters (4.84 M) while sustaining a high-throughput inference speed of 139.00 FPS. This research provides a robust and efficient foundational framework for intelligent yield estimation and precision orchard management.