Visible-infrared (RGB-IR) collaborative perception is essential for robust multispectral object detection under challenging illumination and aerial scenarios.However, RGB and IR modalities inherently exhibit asymmetric reliability and information density across different physical environments. Although existing methods alleviate cross-modal inconsistency through attention mechanisms or feature alignment strategies, many of them still lack explicit reliability-conditioned role assignment, making it difficult to adaptively handle asymmetric information across modalities.To address these limitations, we propose the Asymmetric Guided Mamba-based Mixture-of-Experts Network (AGM$^2$Net), a context-aware expert scheduling framework that reformulates RGB-IR fusion as reliability-aware asymmetric information allocation, where learned routing distributions provide an interpretable proxy for modality-dependent expert preferences.For intra-modal processing, we propose the Adaptive Wavelet Enhanced Downsampling (AWEDown) module, which performs asymmetric frequency purification by selectively suppressing redundant low-frequency structural responses while preserving informative high-frequency details.For inter-modal interaction, we design the MambaMoE module, where heterogeneous experts and sparse routing dynamically select spatial, global, and directional query-modulate interaction policies for reliability-aware cross-modal refinement.Furthermore, to alleviate semantic-spatial inconsistency during multi-scale integration, we introduce the Asymmetric Semantic-Guided Fusion (ASGFusion) module, where semantically rich deep features selectively guide the filtering and refinement of shallow spatial details.Experiments on LLVIP, FLIR-aligned, and DroneVehicle show that AGM$^2$Net achieves state-of-the-art results, with 97.6\%, 82.6\%, and 85.2\% $mAP_{0.5}$, respectively.The source code is available at \href{https://github.com/HZAI-ZJNU/AGM2Net}{https://github.com/HZAI-ZJNU/AGM2Net}.