Deploying robust ship detectors in real-world maritime environments is severely bottlenecked by the dual challenges of strictly constrained computational resources and complex background interferences, such as dense berthing, wake patterns, and SAR speckle. To solve these problems, we propose ARGO-Net, a highly efficient architecture tailored for multi-modal maritime detection. At its core, ARGO-Net extracts physically meaningful and highly discriminative features through three targeted innovations. First, a Background Suppression and Reconstruction Module (BSRM) is developed to mitigate irregular coastal clutter and speckle in the frequency domain, reconstructing resilient spatial representations. Second, to capture the intrinsic morphological properties of ships, the High-Resolution Preserving Feature Network (HRPFN) employs geometry-oriented strip convolutions alongside an adaptive scale mechanism, effectively preserving the structural continuity of elongated hulls across extreme scale variations. Finally, a Semantic–Detail Alignment Fusion (SDAF) module is introduced to resolve cross-level spatial mismatches, ensuring that deep semantic context precisely informs low-level boundary localization. Extensive evaluations on the SeaShips and SSDD benchmarks highlight the exceptional efficiency–accuracy balance of ARGO-Net. With a marginal footprint of merely 2.3 M parameters and 6.9 G FLOPs, ARGO-Net achieves 97.9%/75.4% (mAP@50/mAP@50:95) on SeaShips and 99.5%/79.3% on SSDD. The proposed framework demonstrates that integrating background-aware feature reconstruction with geometry-driven fusion yields state-of-the-art localization precision without compromising lightweight deployability.