Single-view height estimation and semantic segmentation have received increasing attention in recent years and play an important role in the photogrammetry and remote sensing communities. The height information and semantic information of images are correlated, and some recent works have shown that multi-task learning methods can achieve complementation of task-related features and improve the prediction results of the multiple tasks. Although much progress has been made in recent works, how to effectively extract and fuse height features and semantic features is still an open issue. In this paper, a self- and cross-enhancement network (SCE-Net) is proposed to jointly perform height estimation and semantic segmentation on single aerial images. A feature separation–fusion module is constructed to effectively separate and fuse height features and semantic features based on an attention mechanism for feature representation enhancement across tasks. In addition, a height-guided feature distance loss and a semantic-guided feature distance loss are designed based on deep metric learning to achieve task-aware feature representation enhancement. Extensive experiments are conducted on the Vaihingen dataset and the Potsdam dataset to verify the effectiveness of the proposed method. The experimental results demonstrate that the proposed SCE-Net could outperform the state-of-the-art methods and achieve better performance in both height estimation and semantic segmentation.

Height estimation from single images, strictly speaking, is an ill-posed problem. However, recently, it is shown that it is both possible and feasible to learn a mapping from image statistics to height information. In spite of recent efforts in this field, how to learn fine-shape preserving features, such as object boundaries and contours, is still an open issue. In this work, we propose a progressive learning network to estimate height information from single aerial images in a coarse-to-fine manner. In particular, a gated feature aggregation module is introduced to effectively combine low-level and high-level features. The proposed method is validated on three public datasets, including the Vaihingen dataset, the Potsdam dataset, and the DFC2019 dataset. Both quantitative and qualitative experimental results demonstrate that the proposed method can achieve more accurate height estimation from single aerial images, especially with better object boundary and contour preserving capability, than four related height estimation methods.