It is a rather practical but difficult task to find a specified target object via autonomous exploration based on natural language descriptions in an unstructured environment. Since the human-annotated data is expensive to gather for the goal-oriented vision-language navigation (GVLN) task, the size of the standard dataset is inadequate, which has significantly limited the accuracy of previous techniques. In this work, we aim to improve the robustness and generalization of the navigator by dynamically providing high-quality pseudo-instructions using a proposed RES-StS paradigm. Specifically, we establish a referring expression speaker (RES) to predict descriptive instructions for the given path to the goal object. Based on an environment-and-object fusion (EOF) module, RES derives spatial representations from the input trajectories, which are subsequently encoded by a number of transformer layers. Additionally, given that the quality of the pseudo labels is important for data augmentation while the limited dataset may also hinder RES learning, we propose to equip RES with a more effective generation ability by using the self-training approach. A trajectory-instruction matching scorer (TIMS) network based on contrastive learning is proposed to selectively use rehearsal of prior knowledge. Finally, all network modules in the system are integrated by suggesting a multi-stage training strategy, allowing them to assist one another and thus enhance performance on the GVLN task. Experimental results demonstrate the effectiveness of our approach. Compared with the SOTA methods, our method improves SR, SPL, and RGS by 4.72%, 2.55%, and 3.45% respectively, on the REVERIE dataset, and 4.58%, 3.75% and 3.14% respectively, on the SOON dataset.

Existing end-to-end depth representation in embodied AI is often task-specific and lacks the benefits of emerging pre-training paradigm due to limited datasets and training techniques for RGB-D videos. To address the challenge of obtaining robust and generalized depth representation for embodied AI, we introduce a unified RGB-D video dataset (UniRGBD) and a novel time-aware contrastive (TAC) pre-training approach. UniRGBD addresses the scarcity of large-scale depth pre-training datasets by providing a comprehensive collection of data from diverse sources in a unified format, enabling convenient data loading and accommodating various data domains. We also design an RGB-Depth alignment evaluation procedure and introduce a novel Near-K accuracy metric to assess the scene understanding capability of the depth encoder. Then, the TAC pre-training approach fills the gap in depth pre-training methods suitable for RGB-D videos by leveraging the intrinsic similarity between temporally proximate frames. TAC incorporates a soft label design that acts as valid label noise, enhancing the depth semantic extraction and promoting diverse and generalized knowledge acquisition. Furthermore, the adjustments in perspective between temporally proximate frames facilitate the extraction of invariant and comprehensive features, enhancing the robustness of the learned depth representation. Additionally, the inclusion of temporal information stabilizes training gradients and enables spatio-temporal depth perception. Comprehensive evaluation of RGB-Depth alignment demonstrates the superiority of our approach over state-of-the-art methods. We also conduct uncertainty analysis and a novel zero-shot experiment to validate the robustness and generalization of the TAC approach. Moreover, our TAC pre-training demonstrates significant performance improvements in various embodied AI tasks, providing compelling evidence of its efficacy across diverse domains.

Existing real-time semantic segmentation models usually pursue lower inference costs by compressing spatial resolution, reducing the number of feature extraction layers, or simplifying the high-resolution generation process. This inevitably leads to huge information loss and ultimately affects segmentation accuracy. To address the problem, a HoloParser is proposed to learn and parse the input image from a holistic perspective of the entire segmentation pipeline. Firstly, in the image preprocessing part, the often-overlooked problem of structural information loss in fast downsampling is highlighted and studied. For this, a non-strided fast downsampling (NFD) module is devised for holo-structure retention. Secondly, for the resolution change processes in segmentation networks, the spatial correlation between up/down-samplings is revisited and focused. And a lossless sample pair (LSP) is proposed for holo-correlation modeling and retrieving. Thirdly, in feature extraction, after re-considering the necessity and redundancy of multiscale learning, a distributed pyramid learning (DPL) module is proposed for lightweight holo-feature extraction with composite receptive fields. Finally, for the high-resolution output generation, a parallel resolution maintenance (PRM) module is put forward via high-low semantics aggregation across stages for holo-semantics utilization. Extensive experiments demonstrate the effectiveness and superiority of the proposed modules. The HoloParser achieves state-of-the-art performance in real-time semantic segmentation, which reports 76.45% mIoU at 280+ fps on the Cityscapes dataset.