Automatic modulation recognition (AMR) is a key technique in modern communications. We propose LM-GDMAF, which integrates lightweight Mamba modules with a GDMAF fusion module. The lightweight Mamba modules efficiently extract temporal features, while GDMAF deeply fuses dual-modality information from IQ time series and frequency-domain spectrograms. On a public dataset, the proposed method improves average recognition accuracy by at least 4.25% compared to baseline methods. The number of computational parameters of the lightweight Mamba module is reduced by approximately 26.1% compared to the standard Mamba model. The complete model achieves average modulation recognition accuracy gains of approximately 11.89% and 14.74% over the respective single-branch models.