The task of Zero-Shot Object Goal Visual Navigation (ZSON) aims to enable robots to locate previously “unseen” objects by visual observations. This task presents a significant challenge since the robot must transfer the navigation policy learned from “seen” objects to “unseen” objects through auxiliary semantic information without training samples, a process known as zero-shot learning. In order to address this challenge, we propose a novel approach termed the Semantic Policy Network (SPNet). The SPNet consists of two modules that are deeply integrated with semantic embeddings: the Semantic Actor Policy (SAP) module and the Semantic Trajectory (ST) module. The SAP module generates actor network weight bias based on semantic embeddings, creating unique navigation policies for different target classes. The ST module records the robot's actions, visual features, and semantic embeddings at each step, and aggregates information in both the spatial and temporal dimensions. To evaluate our approach, we conducted extensive experiments using MP3D dataset, HM3D dataset and RoboTHOR. Experimental results indicate that the proposed method outperforms other ZSON methods for both seen and unseen target classes.
30 November 2023 in IEEE Transactions on Intelligent Vehicles
Source-free domain adaptive object detection (source-free DAOD) seeks to adapt a detector pre-trained on a source domain to an unlabeled target domain without requiring access to annotated source domain data. To address challenges posed by domain shifts, current source-free DAOD approaches mainly rely on the self-training paradigm, where pseudo labels are predicted and employed to fine-tune the detector on unlabeled target domain. However, these methods often encounter issues related to intra-class variation, resulting in category-specific biases and noisy pseudo labels. In response, we present an effective Multi-Prototype Guided source-free DAOD method, dubbed MPG, consisting of two key components: multi-prototype guided pseudo labeling (MPPL) and multi-prototype guided consistency regularization (MPCR) modules. In the MPPL module, we construct category-specific multiple prototypes to better represent the category with intra-class variations. Specifically, multiple prototypes with adaptive cluster centroids are introduced for each category to effectively capture the intra-class variations. Through the implementation of the proposed MPPL module, we derive more accurate pseudo labels by assessing the proximity of instance features to multiple category prototypes. In the MPCR module, we introduce multi-level consistency regularization, including prototype-based consistency and prediction consistency, which encourages the model to overlook style perturbations and learn domain-invariant representations. Extensive experiments on five public driving datasets demonstrate that MPG outperforms existing state-of-the-art methods, showcasing its effectiveness in adapting object detectors to target domains.
23 November 2023 in IEEE Transactions on Pattern Analysis and Machine Intelligence
This article targets the task of novel category discovery (NCD), which aims to discover unknown categories when a certain number of classes are already known. The NCD task is challenging due to its closeness to real-world scenarios, where we have only encountered some partial classes and corresponding images. Unlike previous approaches to NCD, we propose a novel adaptive prototype learning method that leverages prototypes to emphasize category discrimination and alleviate the issue of missing annotations for novel classes. Concretely, the proposed method consists of two main stages: prototypical representation learning and prototypical self-training. In the first stage, we develop a robust feature extractor that could effectively handle images from both base and novel categories. This ability of instance and category discrimination of the feature extractor is boosted by self-supervised learning and adaptive prototypes. In the second stage, we utilize the prototypes again to rectify offline pseudo labels and train a final parametric classifier for category clustering. We conduct extensive experiments on four benchmark datasets, demonstrating our method’s effectiveness and robustness with state-of-the-art performance.
12 July 2023 in Computer Vision and Image Understanding