Video moment localization has gradually become a hot research problem in video understanding. Despite much remarkable progress, there are still two limitations in the following aspects. Firstly, previous methods usually directly regarded the moment candidate with the highest confidence score as the final localization result; they overlooked the inevitable deviations between moment candidates and the ground truth. Secondly, past models have not considered the problem that moment candidates with various qualities have different impacts on model training. Therefore, this paper proposes a novel Temporal Map-based Boundary Refinement Network to solve the above problems. Specifically, besides the conventional confidence scores’ prediction network, we introduce a boundary refinement network based on the 2D temporal map, which can fine-tune the temporal boundaries of generated moment candidates to obtain more precise results. Additionally, to discriminately treat diverse moment candidates with different qualities and further boost the localization performance, we devise an innovative weighted refinement loss to guide the model to focus more on refining processes of those moment candidates closer to the ground truth. Finally, we evaluate our model on two publicly available datasets, and the results of extensive experiments show that our technique outperforms the state-of-the-art methods (e.g., for the ActivityNet Captions dataset, a relative improvement of 4.63% on R1@0.7).

Few-shot instance segmentation (FSIS) is devised to address the intricate challenge of instance segmentation when labeled data for novel classes is scant. Nevertheless, existing methodologies encounter notable constraints in the agile expansion of novel classes and the management of memory overhead. The integration workflow for novel classes is inflexible, and given the necessity of retaining class exemplars during both training and inference stages, considerable memory consumption ensues. To surmount these challenges, this study introduces an innovative framework encompassing a two-stage “base training-novel class fine-tuning” paradigm. It acquires discriminative instance-level embedding representations. Concretely, instance embeddings are aggregated into class prototypes, and the storage of embedding vectors as opposed to images inherently mitigates the issue of memory overload. Via a Region of Interest (RoI)-level cosine similarity matching mechanism, the flexible augmentation of novel classes is realized, devoid of the requirement for supplementary training and independent of historical data. Experimental validations attest that this approach significantly outperforms state-of-the-art techniques in mainstream benchmark evaluations. More crucially, its memory-optimized attributes facilitate, for the first time, the conjoint assessment of FSIS performance across all classes within the COCO dataset. Visualized instances (incorporating colored masks and class annotations of objects across diverse scenarios) further substantiate the efficacy of the method in real-world complex contexts.

This paper introduces a novel framework for unified incremental few-shot object detection (iFSOD) and instance segmentation (iFSIS) using the Transformer architecture. Our goal is to create an optimal solution for situations where only a few examples of novel object classes are available, with no access to training data for base or old classes, while maintaining high performance across both base and novel classes. To achieve this, We extend Mask-DINO into a two-stage incremental learning framework. Stage 1 focuses on optimizing the model using the base dataset, while Stage 2 involves fine-tuning the model on novel classes. Besides, we incorporate a classifier selection strategy that assigns appropriate classifiers to the encoder and decoder according to their distinct functions. Empirical evidence indicates that this approach effectively mitigates the over-fitting on novel classes learning. Furthermore, we implement knowledge distillation to prevent catastrophic forgetting of base classes. Comprehensive evaluations on the COCO and LVIS datasets for both iFSIS and iFSOD tasks demonstrate that our method significantly outperforms state-of-the-art approaches.