7 June 2021 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Designing effective architectures is one of the key factors behind the success of deep neural networks. Existing deep architectures are either manually designed or automatically searched by some Neural Architecture Search (NAS) methods. However, even a well-designed/searched architecture may still contain many nonsignificant or redundant modules/operations (e.g., some intermediate convolution or pooling layers). Such redundancy may not only incur substantial memory consumption and computational cost but also deteriorate the performance. Thus, it is necessary to optimize the operations inside an architecture to improve the performance without introducing extra computational cost. To this end, we have proposed a Neural Architecture Transformer (NAT) method which casts the optimization problem into a Markov Decision Process (MDP) and seeks to replace the redundant operations with more efficient operations, such as skip or null connection. Note that NAT only considers a small number of possible replacements/transitions and thus comes with a limited search space. As a result, such a small search space may hamper the performance of architecture optimization. To address this issue, we propose a Neural Architecture Transformer++ (NAT++) method which further enlarges the set of candidate transitions to improve the performance of architecture optimization. Specifically, we present a two-level transition rule to obtain valid transitions, i.e., allowing operations to have more efficient types (e.g., convolution ${\to }$ separable convolution) or smaller kernel sizes (e.g., $5{\times }5 {\to } 3{\times }3$ ). Note that different operations may have different valid transitions. We further propose a Binary-Masked Softmax (BMSoftmax) layer to omit the possible invalid transitions. Last, based on the MDP formulation, we apply policy gradient to learn an optimal policy, which will be used to infer the optimized architectures. Extensive experiments show that the transformed architectures significantly outperform both their original counterparts and the architectures optimized by existing methods.
30 September 2014 in International Journal of Computer Vision
20 September 2019 in IEEE Transactions on Image Processing
The human visual system tends to consider saliency of an object as a whole. Some object-level saliency detection methods have been proposed by leveraging object proposals in bounding boxes, and regarding the entire bounding box as one candidate salient region. However, the bounding boxes can not provide exact object position and a lot of pixels in bounding boxes belong to the background. Consequently, background pixels in bounding box also show high saliency. Besides, acquiring object proposals needs high time cost. In order to compute object-level saliency, we consider region growing from some seed superpixels, to find one surrounding region which probably represents the whole object. The desired surrounding region has similar appearance inside and obvious difference with the outside, which is proposed as maximally stable region (MSR) in this paper. In addition, one effective seed superpixel selection strategy is presented to improve speed. MSR based saliency detection is more robust than pixel or superpixel level methods and object proposal based methods. The proposed method significantly outperforms the state-of-the-art unsupervised methods at 50 FPS. Compared with deep learning based methods, we show worse performance, but with about 1200-1600 times faster, which means better trade-off between performance and speed.
12 December 2022 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Neural architecture search (NAS) can automatically discover well-performing architectures in a large search space and has been shown to bring improvements to various applications. However, the computational burden of NAS is huge, since exploring a large search space can need evaluating more than thousands of architecture samples. To improve the sample efficiency of search space exploration, predictor-based NAS methods learn a performance predictor of architectures, and utilize the predictor to sample worth-evaluating architectures. The encoding scheme of NN architectures is crucial to the predictor's generalization ability, and thus crucial to the efficacy of the NAS process. To this end, we have designed a generic Graph-based neural ArchiTecture Encoding Scheme (GATES), a more reasonable modeling of NN architectures that mimics their data processing. Nevertheless, GATES is unaware of the concrete computing semantic of NN operations or architectures. Thus, the learning of operation embeddings and weights in GATES can only exploit the information in architectures-performance pairs. We propose GATES++, which incorporates multifaceted information about NN's operation-level and architecture-level computing semantics into its construction and training, respectively. Experiments on benchmark search spaces show that both the operation-level and architecture-level information can bring improvements alone, and GATES++ can discover better architectures after evaluating the same number of architectures.