Semantic segmentation of Mars scenes has a crucial role in Mars rovers science missions. Current CNN-based composition of U-Net has powerful information extraction capabilities; however, convolutional localization suffers from the limited global context modeling capability. Although Transformer global modeling has performed well, it still encounters obstacles in the extraction and retention of low-level features. This issue is particularly relevant for Martian rocks with their varying shapes, textures, and sizes in Mars scenes. In this paper, we propose a novel Transformer semantic segmentation framework for Martian rock images, called MarsFormer, that consists of an encoder-decoder structure connected through a Feature Enhancement Module (FEM) and a Window Transformer Block (WTB). Specifically, multi-scale hierarchical features are generated by the Mix Transformer encoders, Upgraded-FFN decoder (UFD) fuses and filters features at different scales, preserving the rich local and global contextual information. FEM enhances the inter-multi-scale feature correlation from both spatial and channel perspectives. WTB captures the long-range contexts and preserves the local features. We built two datasets of synthetic and real Martian rocks. The synthetic dataset is SynMars, referencing data from the ZhuRong rover taken from its Virtual Terrain Engine. The other dataset is MarsData-V2, from real Mars scenes, and published recently in our previous study. Extensive experiments conducted on both datasets showed that MarsFormer achieves superiority in Martian rock segmentation, obtaining state-of-the-art performance with favorable computational simplicity. The data is available at: https://github.com/CVIR-Lab/SynMars.

Martian rock segmentation aims to separate rock pixels from background, which plays a crucial role in downstream tasks, such as traversing and geologic analysis by Mars rovers. The U-Nets have achieved certain results in rock segmentation. However, due to the inherent locality of convolution operations, U-Nets are inadequate in modeling global context and long-range spatial dependencies. Although emerging Transformers can solve this, they suffer from difficulties in extracting and retaining sufficient low-level local information. These shortcomings limit the performance of existing networks for Martian rocks that are variable in shape, size, texture and color. Therefore, we propose RockFormer, the first U-shaped Transformer framework for Mars rock segmentation, consisting of a hierarchical encoder-decoder architecture with a Feature Refining Module (FRM) connected between them. Specifically, the encoder hierarchically generates multi-scale features using an improved vision Transformer (improved-ViT), where both abundant local information and long-range contexts are exploited. The FRM removes less representative features and captures global dependencies between multi-scale features, improving RockFormer’s robustness to Martian rocks with diverse appearances. The decoder is responsible for aggregating these features for pixel-wise rock prediction. For evaluation, we establish two Mars rock datasets, including both real and synthesized images. One is MarsData-V2, an extension of our previously published MarsData collected from real Mars rocks. The other is SynMars, a synthetic dataset sequentially photographed from a virtual terrain built referring to the TianWen-1 dataset. Extensive experiments on the two datasets show the superiority of RockFormer for Martian rock segmentation, achieving state-of-the-art performance with decent computational simplicity.

Martian surface, as a typical unstructured terrain, is extremely challenging for Mars exploration missions. Commonly, Mars rovers require multiple sensors to explore such harsh environment, such as depth cameras, range finder and other devices. However, the onboard load, power and storage of rovers are not sufficient to achieve high-level stereoscopic perception, which can be adverse to downstream tasks such as visual navigation and scientific exploration. To this end, in this paper we propose a high-level awareness perception light-weight framework using only close-shot monocular images to implement semantic 3D reconstruction of Martian landforms. This framework consists of two parts. One is a semantic segmentation module based on the proposed real-time Mars terrain segmentation (RMTS) network to extract intra-class and inter-class contexts by local supervision. The other is a depth generation module based on a dual-encoder pix2pix network to encode the visual and semantic information of monocular images, simultaneously. To validate the proposed framework, we construct a Martian planar-stereo dataset based on AI4Mars, an open-source semantic segmentation dataset for Mars surface. It contains monocular close-up Martian images, semantic images and depth images that match each other. After training, the accuracy of proposed semantic segmentation model can reach 84.0% mIoU, with 152.2 FPS on a single RTX6000-24GB GPU. The absolute relative error of pixels in depth images between generation model and the ground truth is 0.367, while the root mean square error gets to 0.510, and the accuracy is 0.753 with 42.9 FPS. The overall environment perception scheme is with 9.5FPS.

This article presents two kernel-based rock detection methods for a Mars rover. Rock detection on planetary surfaces is particularly pivotal for planetary vehicles regarding navigation and obstacle avoidance. However, the diverse morphologies of Martian rocks, the sparsity of pixel-wise features, and engineering constraints are great challenges to current pixel-wise object detection methods, resulting in inaccurate and delayed object location and recognition. We therefore propose a region-wise rock detection framework and design two detection algorithms, kernel principle component analysis (KPCA)-based rock detection (KPRD) and kernel low-rank representation (KLRR)-based rock detection (KLRD), using hypotheses of feature and sub-spatial separability. KPRD is based on KPCA and is expert in real-time detection yet with less accurate performance. KLRD is based on KPRD with KLRR which can generate more precise rock detection results with less delay. To validate the efficiency of the proposed methods, we build a small-scale Martian rock dataset, MarsData, containing various rocks. Preliminary experimental results show that our methods are efficient in dealing with complex images containing rocks, shadows, and gravel. The code and data are available at: https://github.com/CVIR-Lab/MarsData.