Multimodal fusion is an essential research area in computer vision application. Though, there are still many obstacles in image fusion domain, that cause the loss key content, due to method limitation or its least efficiency. To solve these problems, a novel method PFAF-Net (Pyramid Feature Attention Fusion strategy) based on multi-scale features with core idea of different level fusion strategy. Firstly, multiscale high-level features with different receptive fields are extracted by pyramid feature extraction module. Secondly, high level and low features are fused with different fusion techniques i.e. attention based fusion strategies are adopted to efficiently fused the global and local features separately. Finally, the fused image is reconstructed with the enhanced fused features by the decoder. Thus, the proposed method not only integrate multimodal data but also has rich content details inferring a superior image fusion. In addition, proposed PFAF-Net has better generalization ability than existing methods on four different multimodal (i.e. multi-modal medical, multi-exposure, multifocus and visual-infrared) of benchmark datasets. Compared with other state-of-the-art fusion methods experimentally, the proposed method has shown comparable or better fusion performance in both subjective and objective evaluation.

Infrared (IR) and visible images are heterogeneous data, and their fusion is one of the important research contents in the remote sensing field. In the last decade, deep networks have been widely used in image fusion due to their ability to preserve high-level semantic information. However, due to the lower resolution of IR images, deep learning-based methods may not be able to retain the salient features of IR images. In this article, a novel IR and visible image fusion based on IR Features & Multiscale Dense Network (IR-MSDNet) is proposed to preserve the content and key target features from both visible and IR images in the fused image. It comprises an encoder, a multiscale decoder, a traditional processing unit, and a fused unit, and can capture incredibly rich background details in visible images and prominent target details in IR features. When the dense and multiscale features are fused, the background details are obtained by utilizing attention strategy, and then combined with complimentary edge features. While IR features are extracted by traditional quadtree decomposition and Bezier interpolation, and further intensified by refinement. Finally, both the decoded multiscale features and IR features are used to reconstruct the final fused image. Experimental evaluation with other state-of-the-art fusion methods validates the superiority of our proposed IR-MSDNet in both subjective and objective evaluation metrics. Additional objective evaluation conducted on the object detection (OD) task further verifies that the proposed IR-MSDNet has greatly enhanced the details in the fused images, which bring the best OD results.

A novel target detection method based on affine invariant interest point detection, feature encoding, and large-margin dimensionality reduction (LDR) is proposed for optical remote sensing images. First, four types of interest point detectors are introduced, and their performance in extracting low-level affine invariant descriptors using affine shape estimation is compared. Such a description can deal with significant affine transformations, including viewpoints. Second, feature encoding, which extends bag-of-words (BOW) by encoding high-order statistics, is selected to generate mid-level representation. Finally, LDR based on the large-margin constraint and stochastic subgradient is introduced to make the high-dimensional mid-level representation applicable for target detection. The experiments on aircraft and vehicle detections illustrate the effectiveness of the affine invariant description and LDR (compared with principal component analysis) in improving the detection performance. The experiments also demonstrate the effectiveness of the proposed method compared with popular approaches including Gabor, HOG, LBP, BOW, and R-CNN.

Remote sensing image scene classification has drawn significant attention for its potential applications in the economy and livelihoods. Unlike the traditional handcrafted features, the convolutional neural networks (CNNs) provides an excellent avenue in obtaining powerful discriminative features. Although tremendous efforts have been made so far in this domain, there are still many open challenges in scene classification due to the scene complexity with higher within-class diversity and between-class similarity. To solve the above-mentioned problems, D- CapsNet is proposed to learn the richer and more robust features for scene classification. It is an end to end network with four types of layers and incorporates visual attention mechanisms. Its diverse capsules encode different properties of complex image scenes, including deep high-level features, spatial attention based on the fusion of multilayers features, both spatial and channel attention based on high-level features, and their fusion. Experiments on three image scene datasets demonstrate that D-CapsNet outperforms other baselines and state-of-the-art methods with a significant improvement in both classification accuracy and speed.

As a sophisticated computing unit, the pyramidal neuron requires sufficient metabolic energy to fuel its powerful computational capabilities. However, the majority of previous works focus on nonlinear integration and energy consumption in individual pyramidal neurons but seldom on the effects of metabolic energy on synaptic transmission and dendritic integration. Here, we developed biologically plausible models to simulate the synaptic transmission and dendritic integration of pyramidal neurons, exploring the relations between synaptic transmission and metabolic energy and between dendritic integration and metabolic energy. We find that synaptic energy not only drives synaptic vesicle cycle, but also participates in the regulation of this cycle. Release probability of synapses adapts to synaptic energy levels by regulating the speed of synaptic vesicle cycle. Besides, we also find that to match neural energy levels, only a part of the synapses receive presynaptic signals during a given period so that neurons have a low action potential frequency. That is, the number of simultaneously active synapses over a period of time should be adapted to neural energy levels.

The extraction of activation vectors (or deep features) from the fully connected layers of a convolutional neural network (CNN) model is widely used for remote sensing image (RSI) representation. In this study, we propose to learn discriminative convolution filter (DCF) based on class-specific separability criteria for linear transformation of deep features. In particular, two types of pretrained CNN called CaffeNet and VGG-VD16 are introduced to illustrate the generality of the proposed DCF. The activation vectors extracted from the fully connected layers of a CNN are rearranged into the form of an image matrix, from which a spatial arrangement of local patches is extracted using sliding window strategy. DCF learning is then performed on each local patch individually to obtain the corresponding discriminative convolution kernel through generalized eigenvalue decomposition. The proposed DCF learning characterizes that a convolutional kernel with small size (e.g., 3 × 3 pixels) can be effectively learned on a small-size local patch (e.g., 8 × 8 pixels), thereby ensuring that the linear transformation of deep features can maintain low computational complexity. Experiments on two RSI datasets demonstrate the effectiveness of DCF in improving the classification performances of deep features without increasing dimensionality.