The development of robust multimodal perception capabilities is critical for embodied agricultural agents to overcome real-world challenges like occlusion and weather variance. This paper proposes SDyFruit-Fuse—a lightweight multimodal AI agent for visible-infrared fusion-based fruit detection. First, a two-branch feature extraction network based on Dypoformer-CNN is proposed. To enhance the representation of global features, we introduce the Swin-Transformer module in the shallow feature extraction part, which captures long-distance dependencies through the self-attention mechanism and is capable of efficiently processing high-resolution images while reducing computational costs and adapting to downstream vision tasks. In the feature extraction segment, this paper utilizes the pooling mechanism within the Dypoformer module to minimize computational load and model parameter counts, thereby significantly improving inference speed. For feature fusion, we design an adaptive cross-modal fusion mechanism dynamically that aligns features to mitigate modality-specific quality variations. Compared to existing multimodal models Compared to existing multimodal image detection models, the model proposed here is only 1.9MB, with an inference time significantly reduced to 0.02 seconds per image. When deployed for kiwifruit detection, it boosts accuracy by 2% and mAP50 by 9.8% with minimal overhead, significantly enhancing robustness in complex environments. This work advances efficient multimodal perception for embodied agricultural agents.