16 June 2022 in IEEE Transactions on Pattern Analysis and Machine Intelligence
An integral part of video analysis and surveillance is temporal activity detection, which means to simultaneously recognize and localize activities in long untrimmed videos. Currently, the most effective methods of temporal activity detection are based on deep learning, and they typically perform very well with large scale annotated videos for training. However, these methods are limited in real applications due to the unavailable videos about certain activity classes and the time-consuming data annotation. To solve this challenging problem, we propose a novel task setting called zero-shot temporal activity detection (ZSTAD), where activities that have never been seen in training still need to be detected. We design an end-to-end deep transferable network TN-ZSTAD as the architecture for this solution. On the one hand, this network utilizes an activity graph transformer to predict a set of activity instances that appear in the video, rather than produces many activity proposals in advance. On the other hand, this network captures the common semantics of seen and unseen activities from their corresponding label embeddings, and it is optimized with an innovative loss function that considers the classification property on seen activities and the transfer property on unseen activities together. Experiments on the THUMOS’14, Charades, and ActivityNet datasets show promising performance in terms of detecting unseen activities.
6 April 2023 in IEEE Transactions on Circuits and Systems for Video Technology
Few-shot semantic segmentation (FSS) has been developed to perform pixel-level segmentation with only a few dense labeled examples for training, which relieves the expensive annotation problem in traditional segmentation models. Current researches on FSS generally act the labeled masks on the corresponding support images to obtain the class-specific embeddings, and predict the pixel-level masks for query images by matching their pixels to these class-specific embeddings. Their performance is difficult to further break through because of the limited supervision from single-view support images and the neglect of position information from similar pixels between query and support images. To solve these issues, we propose a novel robust prior mask guided model named RPMG-FSS for the challenging FSS task. The core of RPMG-FSS is to produce a robust prior mask with good generalization ability on novel classes to better assist the following query mask prediction. Note that each element in the prior mask corresponds to one pixel in query image. It not only considers the interaction within one view and between multiple views of the support image, but also fuses the top- k similarity values to all support pixels and these pixels’ position information. The parameters in RPMG-FSS are optimized with the combination of segmentation loss and multi-view contrastive loss. Comprehensive experiments on two datasets show that our RPMG-FSS achieves outstanding performance comparing with the current popular baselines. The code is released on https://github.com/dxzxy12138/RPMG-FSS/tree/master.
26 July 2019 in IEEE Transactions on Multimedia
Image-sentence matching is a challenging task for the heterogeneity-gap between different modalities. Ranking-based methods have achieved excellent performance on this task in the past decades. Given an image query, these methods typically assume that the correct matched image-sentence pair must rank before all the other mismatched ones. However, this assumption may be too strict and prone to the overfitting problem, especially when some sentences in a massive database are similar and confusable with one another. In this paper, we relax the traditional ranking loss and propose a novel deep multi-modal network with a top- $k$ ranking loss to mitigate the data ambiguity problem. With this strategy, query results will not be penalized unless the index of ground truth is outside the range of top- $k$ query results. Considering the non-smoothness and non-convexity of the initial top- $k$ ranking loss, we exploit a tight convex upper bound to approximate the loss and then utilize the traditional back-propagation algorithm to optimize the deep multi-modal network. Finally, we apply the method on three benchmark datasets, namely, Flickr8k, Flickr30k, and MSCOCO. Empirical results on metrics R@K(K=1,5,10) show that our method achieves comparable performance to state-of-the-art methods.
29 April 2021 in IEEE Transactions on Circuits and Systems for Video Technology
Prevailing deep methods for image recognition require massive labeled samples in each visual category for training. However, large amounts of data annotations are timeconsuming, and some uncommon categories only have rare samples available. For this issue, we focus on more challenging few-shot learning (FSL) task, where just few labeled images are used in the training stage. Existing FSL models are constructed with various convolutional neural networks (CNNs), which are trained on an auxiliary base dataset and evaluated for new fewshot predictions on a novel dataset. The performance of these models is difficult to break through because of the domain shift between base and novel datasets and the monotonous network architectures. Considering that, we propose a novel automatic attribute consistent network called Auto-ACNet to overcome the above problems. On one hand, Auto-ACNet utilizes the attribute information about base and novel categories to guide the procedure of representation learning. It introduces the consistent and non-consistent subnets to capture the common and different attributes of image pair, which helps to mitigate the domain shift problem. On the other hand, the architecture of Auto-ACNet is searched with the popular neural architecture search (NAS) technique DARTS, for obtaining a superior FSL network automatically. And the DARTS’s search space is improved by adding the position-aware module to extract the attribute characteristics better. Extensive experimental results on two datasets indicate that the proposed Auto-ACNet achieves significant improvement over the state-of-the-art competitors in this literature.
31 December 2021 in IEEE Transactions on Knowledge and Data Engineering
Currently, the non-transparent computing process of deep learning has become a significant reason hindering its further development. The Neural-Symbolic (NS) system formed by integrating logic rules into neural networks has attracted increasing attention owing to its direct interpretability. Embedding symbolic logical formulas into a low-dimensional continuous space provides an effective way for the NS system. However, current studies are all constrained by the modeling ability for its syntactic structure and fail to preserve the intrinsic semantics in embeddings, which causes poor performance on downstream reasoning tasks. To this end, this paper proposes a novel method of Con trastive G raph R epresentations (ConGR) for logical formulas embedding. First, to improve the modeling ability for the syntactic structure, ConGR introduces a densely connected graph convolutional network (GCN) with an attention mechanism to process syntax parsing graphs of formulas. In this way, discriminative local and global embeddings of formulas are obtained at the syntax level. Second, the contrastive instances (positive or negative) for each anchor formula are generated by the transformation under the guidance of logical properties. To preserve semantic information, two types of contrast, global-local and global-global, are carried out to refine formula embeddings. Extensive experiments demonstrate that ConGR obtains superior performance against state-of-the-art baselines on entailment checking and premise selection datasets.