Deep neural networks (DNNs) are vulnerable to adversarial examples where inputs with imperceptible perturbations mislead DNNs to incorrect results. Despite the potential risk they bring, adversarial examples are also valuable for providing insights into the weakness and blind-spots of DNNs. Thus, the interpretability of a DNN in the adversarial setting aims to explain the rationale behind its decision-making process and makes deeper understanding which results in better practical applications. To address this issue, we try to explain adversarial robustness for deep models from a new perspective of neuron sensitivity which is measured by neuron behavior variation intensity against benign and adversarial examples. In this paper, we first draw the close connection between adversarial robustness and neuron sensitivities, as sensitive neurons make the most non-trivial contributions to model predictions in the adversarial setting. Based on that, we further propose to improve adversarial robustness by stabilizing the behaviors of sensitive neurons. Moreover, we demonstrate that state-of-the-art adversarial training methods improve model robustness by reducing neuron sensitivities, which in turn confirms the strong connections between adversarial robustness and neuron sensitivity. Extensive experiments on various datasets demonstrate that our algorithm effectively achieves excellent results. To the best of our knowledge, we are the first to study adversarial robustness using neuron sensitivities.

Open World Object Detection (OWOD), simulating the real dynamic world where knowledge grows continuously, attempts to detect both known and unknown classes and incrementally learn the identified unknown ones. Recently a few studies have introduced and explored the OWOD problem, however, the main challenges in the OWOD task that distinguishing unknown classes from the background (Unknown Objectness) or known classes (Unknown Discrimination) have not been well solved, and there is lacking systematic analysis of benchmark and metrics for evaluating the OWOD task. In this paper, we revisit the OWOD problem and rethink it from benchmark, metrics, and algorithm perspectives. First, we propose five fundamental benchmark principles in line with the OWOD definition and construct two OWOD benchmarks according to the principles for a fair evaluation. Second, we point out that existing metrics neglect the detection performance of unknown classes and further design two additional metrics specific to the OWOD problem, filling the void of evaluating from the perspective of unknown classes. Finally, we introduce a novel and effective OWOD framework with an auxiliary Proposal ADvisor (PAD) and a Class-specific Expelling Classifier (CEC). The non-parametric PAD improves Unknown Objectness by assisting RPN in identifying more accurate unknown proposals based on the class-agnostic property of the object and aggregation through spatial and appearance similarity, while CEC enhances the Unknown Discrimination by calibrating the over-confident activation boundary and suppressing confusing predictions through a class-specific expelling function. Comprehensive experiments conducted on both fair benchmarks based on our OWOD benchmark principles and the original benchmark demonstrate that our method outperforms other state-of-the-art object detection methods in terms of both existing and our new metrics.

Recently, deep neural networks have achieved promising performance for in-filling large missing regions in image inpainting tasks. They have usually adopted the standard convolutional architecture over the corrupted image, leading to meaningless contents, such as color discrepancy, blur, and other artifacts. Moreover, most inpainting approaches cannot handle well the case of a large contiguous missing area. To address these problems, we propose a generic inpainting framework capable of handling incomplete images with both contiguous and discontiguous large missing areas. We pose this in an adversarial manner, deploying regionwise operations in both the generator and discriminator to separately handle the different types of regions, namely, existing regions and missing ones. Moreover, a correlation loss is introduced to capture the nonlocal correlations between different patches, and thus, guide the generator to obtain more information during inference. With the help of regionwise generative adversarial mechanism, our framework can restore semantically reasonable and visually realistic images for both discontiguous and contiguous large missing areas. Extensive experiments on three widely used datasets for image inpainting task have been conducted, and both qualitative and quantitative experimental results demonstrate that the proposed model significantly outperforms the state-of-the-art approaches, on the large contiguous and discontiguous missing areas.

Action Recognition is a fundamental task in computer vision field, with a wide range of applications in autonomous driving, security monitoring, etc. However, previous action recognition approaches usually suffer from the inappropriate spatio-temporal modeling or high computational consumption (e.g., 3D CNN). In this paper, we propose a novel Spatio-Temporal Adaptive Network (STANet) with bidirectional temporal difference, consisting of a Temporal Adaptive module (TA) and a Spatial Adaptive (SA) module, to sufficiently extract the crucial motion information and model the spatial pivotal appearance information from both forward and backward perspectives, respectively. Specifically, the Temporal Adaptive module uses bidirectional temporal differences to learn valuable motion trends and balance the static semantics and dynamic motion for a certain action during information fusion; while the Spatial Adaptive module uses the bidirectional temporal difference to obtain the spatio-channel attention to stress the discriminative position-relevant and semantic-relevant appearance features. Extensive experiments conducted on widely-used action recognition benchmarks UCF-101, HMDB-51, Something-Something V1, and Kinetics-400 prove the effectiveness of the proposed methods compared to other state-of-the-art approaches.

Few-shot learning, aiming to learn novel concepts from one or a few labeled examples, is an interesting and very challenging problem with many practical advantages. Existing few-shot methods usually utilize data of the same classes to train the feature embedding module and in a row, which is unable to learn adapting to new tasks. Besides, traditional few-shot models fail to take advantage of the valuable relations of the support-query pairs, leading to performance degradation. In this article, we propose a transductive relation-propagation graph neural network (GNN) with a decoupling training strategy (TRPN-D) to explicitly model and propagate such relations across support-query pairs, and empower the few-shot module the ability of transferring past knowledge to new tasks via the decoupling training. Our few-shot module, namely TRPN, treats the relation of each support-query pair as a graph node, named relational node, and resorts to the known relations between support samples, including both intraclass commonality and interclass uniqueness. Through relation propagation, the model could generate the discriminative relation embeddings for support-query pairs. To the best of our knowledge, this is the first work that decouples the training of the embedding network and the few-shot graph module with different tasks, which might offer a new way to solve the few-shot learning problem. Extensive experiments conducted on several benchmark datasets demonstrate that our method can significantly outperform a variety of state-of-the-art few-shot learning methods.

Recently, few-shot object detection (FSOD) has become an increasing research focus, which can largely alleviate the heavy dependency on expensive annotations in the traditional object detection task. However, existing FSOD approaches fail to generate sufficient high-quality positive region proposals which are the key to detection performance, due to the lack of informative knowledge from base classes and non-specific alteration for novel classes. To address the problem, this paper presents a simple yet effective few-shot object detection framework referred to as Temporal Speciation Network (TeSNet) with an evolving training, which improves the diversity and rationality of positive proposal generation. Our TeSNet, imitating the natural evolution which relies on inheritation and mutation, correspondingly consists of two key components: a Selective Recombination Module (SRM) for effectively inheriting from base classes and a Mutational Region Proposal Network (MRPN) for flexibly mutating according to the unique traits of novel samples. Specifically, SRM selects and reorganizes relevant base categories, and further instantiates diverse individuals to ensure the diversity of positive proposals. MRPN adapts the parameters trained on base classes aiming for accurately locating positive proposals. Extensive experiments are conducted on several commonly-used datasets, in which our TeSNet achieves state-of-the-art results and outperforms baselines by large margin.