30 December 2024 in IEEE Transactions on Multimedia
Recent progress in remote sensing image (RSI) super-resolution (SR) has exhibited remarkable performance using deep neural networks, e.g., Convolutional Neural Networks and Transformers. However, existing SR methods often suffer from either a limited receptive field or quadratic computational overhead, resulting in sub-optimal global representation and unacceptable computational costs in large-scale RSI. To alleviate these issues, we develop the first attempt to integrate the Vision State Space Model (Mamba) for RSI-SR, which specializes in processing large-scale RSI by capturing long-range dependency with linear complexity. To achieve better SR reconstruction, building upon Mamba, we devise a Frequency-assisted Mamba framework, dubbed FMSR, to explore the spatial and frequent correlations. In particular, our FMSR features a multi-level fusion architecture equipped with the Frequency Selection Module (FSM), Vision State Space Module (VSSM), and Hybrid Gate Module (HGM) to grasp their merits for effective spatial-frequency fusion. Considering that global and local dependencies are complementary and both beneficial for SR, we further recalibrate these multi-level features for accurate feature fusion via learnable scaling adaptors. Extensive experiments on AID, DOTA, and DIOR benchmarks demonstrate that our FMSR outperforms state-of-the-art Transformer-based methods HAT-L in terms of PSNR by 0.11 dB on average, while consuming only 28.05% and 19.08% of its memory consumption and complexity, respectively. Code will be available at https://github.com/XY-boy/FreMamba
7 March 2025 in IEEE Transactions on Image Processing
How to aggregate spatial-temporal information plays an essential role in video super-resolution (VSR) tasks. Despite the remarkable success, existing methods adopt static convolution to encode spatial-temporal information, which lacks flexibility in aggregating information in large-scale remote sensing scenes, as they often contain heterogeneous features (e.g., diverse textures). In this paper, we propose a spatial feature diversity enhancement module (SDE) and channel diversity exploration module (CDE), which explore the diverse representation of different local patterns while aggregating the global response with compactly channel-wise embedding representation. Specifically, SDE introduces multiple learnable filters to extract representative spatial variants and encodes them to generate a dynamic kernel for enriched spatial representation. To explore the diversity in the channel dimension, CDE exploits the discrete cosine transform to transform the feature into the frequency domain. This enriches the channel representation while mitigating massive frequency loss caused by pooling operation. Based on SDE and CDE, we further devise a multi-axis feature diversity enhancement (MADE) module to harmonize the spatial, channel, and pixel-wise features for diverse feature fusion. These elaborate strategies form a novel network for satellite VSR, termed MADNet, which achieves favorable performance against state-of-the-art method BasicVSR++ in terms of average PSNR by 0.14 dB on various video satellites, including JiLin-1, Carbonite-2, SkySat-1, and UrtheCast. Code will be available at https://github.com/XY-boy/MADNet.
28 December 2023 in IEEE Transactions on Geoscience and Remote Sensing
Hyperspectral (HS) video is able to capture abundant spectral, spatial, and temporal information of objects, which overcomes the limitations of common red-green-blue (RGB) video in complex scenarios such as similar appearances and background clutters. However, most of trackers apply hand-crafted features extracted from manually selected bands instead of deep features for object representations due to limited HS data and the band gap problem. Each HS image consists of many bands, and it is challenging to fully interact with the band information while maintaining tracking speed. To this end, this paper proposes a novel end-to-end spectral awareness interaction network with dynamic template (SPIRIT) for HS video object tracking. Firstly, a spectral awareness module is proposed to learn band contributions with consideration of nonlinear and global interactions between HS bands. It can also cooperate with the feature extraction module pre-trained with RGB data to attenuate the band gap and data-hungry. Secondly, an interaction module is proposed to achieve inter- and intra-band feature interactions to enhance tracking performance while improving efficiency. Furthermore, the proposed method contains a novel update module that evaluates the tracking confidence of the current state to adapt to object changes and attenuate tracking drifts. Extensive experiments demonstrate the superiority of our approach compared to state-of-the-arts while meeting real-time demands.
20 September 2024 in IEEE Transactions on Geoscience and Remote Sensing
Hyperspectral image (HSI) clustering is a fundamental yet challenging task that groups image pixels with similar features into distinct clusters. Among various approaches, contrastive learning methods, which employ the concept of encouraging semantically similar samples to move closer together while pushing semantically inconsistent samples apart, have garnered significant attention due to their promising performance. However, the most prevalent approaches face two major limitations: 1) treating all samples indiscriminately during optimization, where the abundance of well-categorized samples overwhelms the feature learning process; 2) tending to introduce noise when constructing positive sample pairs through view augmentation or searching the nearest neighbors, which would cause semantic drift of sample features. To solve these issues, we propose a graph autoencoder-based deep clustering framework named SSGCC that constructs spatial-spectral dual views without data augmentation and focuses more on hard samples rather than treating all samples equally with the aid of spatial-spectral features. Concretely, we extract the spectral features and the neighborhood spatial features of the samples as dual branches to avoid the noise caused by data augmentation, and develop the cluster-oriented consistency learning to facilitate the exchange of knowledge between the two spectral-spatial perspectives. Additionally, we propose a hard sample mining-based contrastive learning scheme with the aid of spatial-spectral features. To better measure the importance of the samples, we combine spatial features and spectral features to calculate the similarity between sample pairs. The weights of hard sample pairs are dynamically up-weight while the easy ones are down-weighting to improve the discriminative capability. Extensive experiments on four benchmark HSI datasets demonstrate the effectiveness and superiority of the proposed methods against state-of-the-art ones.
22 June 2022 in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
Satellite video (SV) can acquire rich spatiotemporal information on the earth. Single object tracking (SOT) in SVs enables the continuous acquisition of the position and range of a specific object, expanding the field of remote-sensing applications. In SVs, objects are small with limited features and vulnerable to tracking drift. In this article, a correlation filter based dual-flow (DF) tracker is proposed to explore how the hybridization of spatial–spectral feature fusion and motion model can boost tracking. To represent small objects, the DF adaptively fuses complementary features using a state-aware indicator in feature flow. In motion flow, the indicator perceives the confidence of the feature flow. A dual-mode prediction model is then constructed to simulate the object's motion pattern and cooperate linear and nonlinear motion patterns to implement SOT in SVs. The ablation experiments demonstrate that the DF contributes to tracking. Experimental comparisons on 14 real SVs captured by the Jilin-1 satellite constellation show that DF achieves optimal performance with an area under the curve of 0.912 in the precision plot, 0.700 in the success plot, and a speed of 155.2 frames per second. This work would encourage the development of remote-sensing ground surveillance.
19 May 2025 in IEEE Transactions on Image Processing
Hyperspectral video (HSV) provides rich spectral-spatial-temporal information, enabling the capture of complex object dynamics beyond the limitations of conventional single-and multi-modal tracking. However, current HSV tracking methods face challenges such as data scarcity, band gaps, spectral fragmentation, temporal underutilization, and high computational load, which constrain performance. In this article, we present SpectralTrack, a novel HSV tracking framework with spectral-spatial fusion and memory enhancement. SpectralTrack incorporates an explicit visual prompting module to mitigate band gaps and spectral fragmentation. We further introduce an extraction-matching-interaction module, which leverages a template-bridging search adapter and a multi-layer perceptron adapter within a multi-modal Transformer architecture for efficient cross-modal feature extraction-matching-interaction. Additionally, a memory perception module enhances state reasoning by injecting temporal prompts to refine spectral and spatial cues. SpectralTrack follows parameter-efficient fine-tuning and feature-level fusion to alleviate data scarcity and reduce computational overhead. We instantiate two variants, SpectralTrack and SpectralTrack+, across nine HSV tracking datasets, demonstrating superior effectiveness over extensive trackers. Implementations and results will be available at https://github.com/YZCU/SpectralTrack.