Indoor-scene semantic segmentation is of great significance to indoor navigation, high-precision map creation, route planning, etc. However, incorporating RGB and HHA images for indoor-scene semantic segmentation is a promising yet challenging task, due to the diversity of textures and structures and the disparity of multi-modality in physical significance. In this paper, we propose a Cross-Modality Attention Network (CMANet) that facilitates the extraction of both RGB and HHA features and enhances the cross-modality feature integration. CMANet is constructed under the encoder–decoder architecture. The encoder consists of two parallel branches that successively extract the latent modality features from RGB and HHA images, respectively. Particularly, a novel self-attention mechanism-based Cross-Modality Refine Gate (CMRG) is presented, which bridges the two branches. More importantly, the CMRG achieves cross-modality feature fusion and produces certain refined aggregated features; it serves as the most crucial part of CMANet. The decoder is a multi-stage up-sampled backbone that is composed of different residual blocks at each up-sampling stage. Furthermore, bi-directional multi-step propagation and pyramid supervision are applied to assist the leaning process. To evaluate the effectiveness and efficiency of the proposed method, extensive experiments are conducted on NYUDv2 and SUN RGB-D datasets. Experimental results demonstrate that our method outperforms the existing ones for indoor semantic-segmentation tasks.

The reconstruction of 3-D models of power pylons from light detection and ranging (LiDAR) data plays an important role in power transmission safety. However, accurate reconstruction of power pylon models still faces challenges, e.g., complex structures, missing data, and occlusion. In this letter, a novel four-component segmentation method is proposed for reconstructing power pylon models. In the proposed method, the pylon components of the pylon head, pylon body, cross-arms, and pedestal are first defined in terms of the common features and functionality of each component. Then, these four components are each segmented and identified based on their position and shape features from the raw point cloud. An improved approach based on Metropolis–Hastings sampling and a simulated annealing algorithm is proposed to estimate the model parameters. Based on the estimated parameters, the 3-D shapes of the individual components are reconstructed and stacked to form the whole pylon model. Experimental results show that our methods are able to reconstruct pylons with complex-shaped heads and multiple cross-arms, with an average reconstruction error of less than 0.3 m. Compared with the standard Metropolis–Hastings algorithm with annealing, the parameter estimation process in our strategy improves the computational efficiency by 7.54%.

Landslide extraction plays a critical role in disaster prevention and mitigation. However, acquiring sufficient landslide samples is often challenging, significantly limiting the performance of existing extraction methods. Self-supervised learning approaches, such as masked autoencoder (MAE) reconstruction and contrastive learning, offer a promising solution by reducing reliance on labeled data. Nonetheless, Transformer-based MAEs are computationally intensive, while CNN-based MAEs often struggle to capture global contextual information. To address these limitations, this letter proposes a CNN-Transformer hybrid masked autoencoder integrated with contrastive learning (CT-MAE) for pretraining a landslide extraction model. HRNet is employed for landslide extraction. An HRNet-based MAE, which has the c-apability to preserve the resolution and effectively integrate both shallow and deep features, is proposed for pretraining the HRNet. Moreover, the traditional Transformer-based MAE is applied as an auxiliary branch, and contrastive learning is utilized on the features extracted from the encoders of both MAEs. This auxiliary branch serves as a form of data augmentation, providing supplementary supervision, and enables the CNN-based MAE to capture richer global features without modifying the underlying CNN architecture. Experimental results on two publicly available datasets demonstrate that the proposed pretraining approach significantly enhances landslide extraction performance, highlighting its potential for large-scale applications.

Employing multimodal remote sensing images (MRSIs) enhances ground object identification, yet the heterogeneity of MRSIs can increase the risk of model overfitting, particularly when training samples are limited. Self-supervised methods, such as masked autoencoder (MAE) reconstruction or contrastive learning, efficiently extract features from MRSIs with minimal reliance on labeled samples. However, current methods have not adequately considered both the shared and complementary features of MRSIs during the pretraining process, limiting feature representation and classification performance. To address these limitations, we propose a self-supervised pretraining framework called shared and complementary information interaction enhancement network (SCIIENet) for hyperspectral images (HSIs) and light detection and ranging (LiDAR)/synthetic aperture radar (SAR) data joint classification. Specifically, the framework incorporates an decoupling-interaction-enhancement approach through a triple-branch MAE architecture. The triple-branch architecture seperately captures the spectral features of HSIs that provide complementary information relative to LiDAR/SAR data, as well as the spatial features of HSIs and LiDAR/SAR data. We also introduce an interaction enhancement strategy that utilizes hierarchical contrastive learning (HCL) to bridge significant gaps in MRSI to emphasize the shared spatial information of MRSIs and incorporates a two-stage cross-modal fusion encoder (TCFE) to integrate the shared and complementary spatial information of different modalities. Moreover, we propose multi-stage skip connections (MSCs) for the reconstruction, which effectively preserve complementary spatial information. Extensive experiments on four datasets show that our method significantly improves classification accuracy compared to several state-of-the-art methods.

Keywords: Indoor Scene, Semantic Segmentation, RGB-D Images, Encode-Decoder process, Fully Convolutional Networks, Multiple Kernel Maximum Mean Discrepancy (MK-MMD), Full Connect CRFs With increasing attention for the indoor environment and the development of low-cost RGB-D sensors, indoor RGB-D images are easily acquired. However, scene semantic segmentation is still an open area, which restricts indoor applications. The depth information can help to distinguish the regions which are difficult to be segmented out from the RGB images with similar color or texture in the indoor scenes. How to utilize the depth information is the key problem of semantic segmentation for RGB-D images. In this paper, we propose an Encode-Decoder Fully Convolutional Networks for RGB-D image classification. We use Multiple Kernel Maximum Mean Discrepancy (MK-MMD) as a distance measure to find common and special features of RGB and D images in the network to enhance performance of classification automatically. To explore better methods of applying MMD, we designed two strategies; the first calculates MMD for each feature map, and the other calculates MMD for whole batch features. Based on the result of classification, we use the full connect CRFs for the semantic segmentation. The experimental results show that our method can achieve a good performance on indoor RGB-D image semantic segmentation.

Weakly supervised semantic segmentation (WSSS) based on image-level labels significantly reduces the labeling burden. However, current mainstream approaches optimize solely using single-image information, neglecting the rich semantic correlation among images and struggling to dynamically suppress interfering information. When confronted with complex backgrounds and multicategory remote sensing (RS) images, intraclass consistency and interclass discrimination pose significant challenges. To address these challenges, this letter proposes the cross-image class token constraints network (CICTC-Net). CICTC-Net establishes semantic correlations across multicategory RS images and implements two modules for targeted optimization. Specifically, the cross-image token enhancement (CITE) module constructs intraclass token relationship graphs and applies cross-image consistency constraints to enhance semantic consistency among objects of the same category. The class-patch interaction refinement (CPIR) module constructs a directed graph of class-patch relationships and employs a neighborhood selection mechanism to refine class tokens, thereby enhancing interclass discriminability. Experiments on two RS datasets demonstrate that this approach significantly outperforms existing state-of-the-art solutions.