Deep convolutional neural networks have greatly enhanced the semantic segmentation of remote sensing images. However, most networks are primarily designed to process imagery with red, green, and blue bands. Although it is feasible to directly utilize established networks and pre-trained models for remotely sensed images, they suffer from imprecise land object contour localization and unsatisfactory segmentation results. These networks still need to explore the domain knowledge embedded in images. Therefore, we boost the segmentation performance of remote sensing images by augmenting the network input with multiple nonlinear spectral indices, such as vegetation and water indices, and introducing a novel holistic attention edge detection network (HAE-RNet). Experiments were conducted on the GID and Vaihingen datasets. The results showed that the NIR-NDWI/DSM-GNDVI-R-G-B (6C-2) band combination produced the best segmentation results for both datasets. The edge extraction block benefits better contour localization. The proposed network achieved a state-of-the-art performance in both the quantitative evaluation and visual inspection.