The joint use of multi-source remote sensing data for earth observation missions has drawn much attention. Although the fusion of several data sources can improve the accuracy of land-cover identification, many technical obstacles exist, such as disparate data structures, irrelevant physical characteristics, and a lack of training data. In this article, a novel dual-branch method, consisting of a hierarchical convolutional neural network (CNN) and a transformer network, is proposed for fusing multi-source heterogeneous information and improving joint classification performance. First, by combining CNN with a transformer, the proposed dual-branch network can significantly capture and learn spectral-spatial features from hyperspectral image data and elevation features from light detection and ranging (LiDAR) data. Then, to fuse these two sets of data features, a cross-token attention fusion encoder is designed in a specialty. The well-designed deep hierarchical architecture takes full advantage of the powerful spatial context information extraction ability of CNN and the strong long-range dependency modeling ability of the transformer network based on self-attention mechanism. Four standard datasets are used in experiments to verify the effectiveness of the approach. The experimental results reveal that the proposed framework can perform noticeably better than state-of-the-art methods. The source code of the proposed method will be available publicly at https://github.com/zgr6010/Fusion_HCT.git.

The low spatial resolution of hyperspectral images leads to the coexistence of multiple ground objects in a single pixel (called mixed pixels). A large number of mixed pixels in a hyperspectral image hinders the subsequent analysis and application of the image. In order to solve this problem, a novel sparse unmixing method, which considers highly similar patches in nonlocal regions of a hyperspectral image, is proposed in this paper. This method exploits spectral correlation by using collaborative sparsity regularization and spatial information by employing total variation and weighted nonlocal low-rank tensor regularization. To effectively utilize the tensor decomposition, nonlocal similar patches are first grouped together. Then, these nonlocal patches are stacked to form a patch group tensor. Finally, weighted low rank tensor regularization is enforced to constrain the patch group to obtain an estimated low rank abundance image. Experiments on simulated and real hyperspectral datasets validated the superiority of the proposed method in better maintaining fine details and obtaining better unmixing results.

The raw video data can be compressed much by the latest video coding standard, high efficiency video coding (HEVC). However, the block-based hybrid coding used in HEVC will incur lots of artifacts in compressed videos, the video quality will be severely influenced. To settle this problem, the in-loop filtering is used in HEVC to eliminate artifacts. Inspired by the success of deep learning, we propose an efficient in-loop filtering algorithm based on the enhanced deep convolutional neural networks (EDCNN) for significantly improving the performance of in-loop filtering in HEVC. Firstly, the problems of traditional convolutional neural networks models, including the normalization method, network learning ability, and loss function, are analyzed. Then, based on the statistical analyses, the EDCNN is proposed for efficiently eliminating the artifacts, which adopts three solutions, including a weighted normalization method, a feature information fusion block, and a precise loss function. Finally, the PSNR enhancement, PSNR smoothness, RD performance, subjective test, and computational complexity/GPU memory consumption are employed as the evaluation criteria, and experimental results show that when compared with the filter in HM16.9, the proposed in-loop filtering algorithm achieves an average of 6.45% BDBR reduction and 0.238 dB BDPSNR gains.

This paper proposes a novel efficient convolutional neural network (CNN) for light field (LF) spatial super-resolution (SR). Due to the high dimensionality of LF data, it is important to decompose the original 4D LF data structure into a lower data subspace helping to reduce memory and computational costs. In this context, we propose a novel hybrid CNN structure working on different 2D data representations to fully handle the 4D structure. Our proposed network can simultaneously exploit information from three different types of 4D LF representations: spatial, angular, and epipolar plane image (EPI) information. Based on pixel similarities of each data subspace, we propose an inter-intra spatial feature extractor (II-SFE) and an inter-intra angular feature extractor (II-AFE) that can exploit the pixel correlations among the spatial-angular domain. Moreover, based on the correlated epipolar geometry among pixels, we also propose a multi-orientation epipolar feature extractor (MO-EFE) by decomposing an LF image into four orientation stacks of views and feeding them into different network branches to learn rich sub-pixel information in the EPI domain. The extracted features of each 2D subspace are integrated by our proposed attention fusion module (AFM), which adaptively rescales the embedding weights based on their correlations. Experimental results on both small- and wide-baseline benchmark LF datasets confirm the effectiveness of the proposed method compared to the well-known state-of-the-art methods in terms of objective and subjective performance.

Content-based Image Retrieval (CBIR) techniques have been extensively studied with the rapid growth of digital images. Generally, CBIR service is quite expensive in computational and storage resources. Thus, it is a good choice to outsource CBIR service to the cloud server that is equipped with enormous recourses. However, the privacy protection becomes a big problem, as the cloud server cannot be fully trusted. In this paper, we propose an outsourced CBIR scheme based on a novel bag-of-encrypted-words (BOEW) model. The image is encrypted by color value substitution, block permutation, and intra-block pixel permutation. Then, the local histograms are calculated from the encrypted image blocks by the cloud server. All the local histograms are clustered together, and the cluster centers are used as the encrypted visual words. In this way, the bag-of-encrypted-words (BOEW) model is built to represent each image by a feature vector, i.e., a normalized histogram of the encrypted visual words. The similarity between images can be directly measured by the Manhattan distance between feature vectors on the cloud server side. Experimental results and security analysis on the proposed scheme demonstrate its search accuracy and security.