To address the challenges of cross-modal feature misalignment and ineffective information fusion caused by the inherent differences in imaging mechanisms, noise statistics, and semantic representations between visible and synthetic aperture radar (SAR) imagery, this paper proposes a multimodal remote sensing object detection method, namely YOLO-CMFM. Built upon the Ultralytics YOLOv11 framework, the proposed approach designs a Cross-Modal Fusion Module (CMFM) that systematically enhances detection accuracy and robustness from the perspectives of modality alignment, feature interaction, and adaptive fusion. Specifically, (1) a Learnable Edge-Guided Attention (LEGA) module is constructed, which leverages a learnable Gaussian saliency prior to achieve edge-oriented cross-modal alignment, effectively mitigating edge-structure mismatches across modalities; (2) a Bidirectional Cross-Attention (BCA) module is developed to enable deep semantic interaction and global contextual aggregation; (3) a Context-Guided Gating (CGG) module is designed to dynamically generate complementary weights based on multimodal source features and global contextual information, thereby achieving adaptive fusion across modalities. Extensive experiments conducted on the OGSOD 1.0 dataset demonstrate that the proposed YOLO-CMFM achieves an $mAP@50$ of 96.2% and an $mAP@50$:95 of 75.1%. While maintaining competitive performance comparable to mainstream approaches at lower IoU thresholds, the proposed method significantly outperforms existing counterparts at high IoU thresholds, highlighting its superior capability in precise object localization. Also, the experimental results on the OSPRC dataset demonstrate that the proposed method can consistently achieve stable gains under different kinds of imaging conditions, including diverse SAR polarizations, spatial resolutions, and cloud occlusion conditions. Moreover, the CMFM can be flexibly integrated into different detection frameworks, which further validates its strong generalization and transferability in multimodal remote sensing object detection tasks.