Significant Wave Height (SWH) is a critical parameter for ocean observation. SWH retrieval using GNSS-R data faces challenges including difficult feature selection, insufficient temporal dependency modeling, and limitations due to single-modality data. This paper proposes a novel Multimodal Transformer Network (MultTransNet) to enhance the accuracy of GNSS-R SWH retrieval. To optimize the feature set, we designed an XGBoost-based iterative feature selection module that effectively eliminates redundant features. To capture complex temporal dependencies and global context, the model employs a Transformer encoder utilizing its self-attention mechanism. Furthermore, to overcome the constraints of single-modality data, we innovatively fused 2D DDM image data with 1D auxiliary parameters, enabling multi-source information integration. Simulation results show that the Transformer architecture reduces Root Mean Square Error (RMSE) by 8.91% and increases Correlation Coefficient (CC) by 4.05% compared to a conventional Deep Neural Network (DNN) model. More significantly, the proposed multimodal algorithm further improves retrieval accuracy by 27.05% (RMSE reduction) and 7.21% (CC increase) compared to its single-modality Transformer counterpart, demonstrating superior performance, especially in complex sea-state conditions.