Waterbody extraction is of great significance for water resource investigation and monitoring. In addition to RGB bands, most common satellite images have a near-infrared (NIR) band. By combining these RGB-NIR bands, certain water, vegetation, and shadow indices can be calculated. The near-infrared band and these indices are very similar to the X modality in RGB+X data (common examples include RGB-D and RGB-Thermal). However, at present, no studies have thoroughly examined multimodal feature fusion from the RGB+X perspective in order to extract waterbodies with high precision. As a result, existing algorithms do not fully utilize satellite image information and have limited generalization ability. To overcome this limitation, we propose a dual-complexity backbone for waterbody extraction from the perspective of RGB+X data semantic segmentation. Its complex Transformer branch is used to extract RGB modality features, while its simple CNN branch is used to extract X modality features. This network structure can effectively capture multimodal, global, and local features in remote sensing images. It can also fully leverage the fact that the scale of RGB image datasets in computer vision is significantly larger than that of remote sensing waterbody extraction datasets. If a large pretrained model is used in the RGB branch, it is unnecessary to freeze the weights. Instead, both branches can be trained jointly, allowing the RGB branch to better adapt to the remote sensing waterbody extraction task without raising concerns that fine-tuning might undermine the pretrained model’s strong representation capability. We also propose two X modality configurations with strong generalization performance. To fully fuse multimodal features, we design a hybrid fusion module combining a CNN and a cross-attention mechanism. To integrate the multi-scale features, we employ a multi-scale Transformer structure in the RGB branch and design a multi-scale decoder. Our algorithm achieves state-of-the-art performance on the GID-5 dataset and competitive performance on the S1S2-Water dataset. Furthermore, it significantly outperforms existing methods in cross-dataset zero-shot transfer between the two datasets, with IoU/F1-score gains of 26.08%/27.33% on GID-5 and 38.74%/31.37% on S1S2-Water over previous SOTA methods. Our processing paradigm of modeling RGB-NIR remote sensing images as RGB+X data shows potential for generalization to other multi-modal remote sensing tasks. The dual-complexity backbone we design also has potential to be extended to other tasks that transfer large pretrained RGB models to remote sensing imagery with RGB-NIR four bands or even more spectral bands. We have open-sourced the code and trained models used in this research.