In the field of hyperspectral video tracking (HVT), occclusion poses a challenging issue without a satisfactory solution. To address this challenge, the current study explores the application of capsule networks in HVT and proposes an approach based on octave convolution and a spatial–spectral capsule network (OCSCNet). Specifically, the spatial–spectral octave convolution module is designed to learn features from hyperspectral images by integrating spatial and spectral information. Hence, unlike traditional convolution, which is limited to learning spatial features, the proposed strategy also focuses on learning and modeling the spectral features. The proposed spatial–spectral capsule network integrates spectral information to distinguish among underlying capsule categories based on their spectral similarity. The approach enhances separability and establishes relationships between different components and targets at various scales. Finally, a confidence threshold judgment module utilizes the information from the initial and adjacent frames for relocating the lost target. Experiments conducted on the HOT2023 dataset illustrate that the proposed model outperforms state-of-the-art methods, achieving a success rate of 65.2% and a precision of 89.3%. In addition, extensive experimental results and visualizations further demonstrate the effectiveness and interpretability of the proposed OCSCNet.

The advent of hyperspectral cameras has popularized the study of hyperspectral video trackers. Although hyperspectral images can better distinguish the targets compared to their RGB counterparts, the occlusion and rotation of the target affect the effectiveness of the target. For instance, occlusion obscures the target, reducing the tracking accuracy and even causing tracking failure. In this regard, this paper proposes a novel hyperspectral video tracker where the double Siamese network (D-Siam) forms the basis of the framework. Moreover, AlexNet serves as the backbone of D-Siam. The current study also adopts a novel spectral–deviation-based dimensionality reduction approach on the learned features to match the input requirements of the AlexNet. It should be noted that the proposed dimensionality reduction method increases the distinction between the target and background. The two response maps, namely the initial response map and the adjacent response map, obtained using the D-Siam network, were fused using an adaptive weight estimation strategy. Finally, a confidence judgment module is proposed to regulate the update for the whole framework. A comparative analysis of the proposed approach with state-of-the-art trackers and an extensive ablation study were conducted on a publicly available benchmark hyperspectral dataset. The results show that the proposed tracker outperforms the existing state-of-the-art approaches against most of the challenges.

In order to address the problems of in-plane rotation and fast motion during near-infrared (NIR) video target tracking, this study explores the application of capsule networks in NIR video and proposes a capsule network method based on background information and spectral position prediction. First, the history frame background information extraction module is proposed. This module performs spectral matching on the history frame images through the average spectral curve of the groundtruth value of the target and makes a rough distinction between the target and the background. On this basis, the background information of history frames is stored as a background pool for subsequent operations. The proposed background target routing module combines the traditional capsule network algorithm with spectral information. Specifically, the similarity between the target capsule and the background capsule in the spectral feature space is calculated, and the capsule weight allocation mechanism is dynamically adjusted. Thus, the discriminative ability of the target and background is strengthened. Finally, the spectral information position prediction module locates the center of the search region in the next frame by fusing the position information and spectral features of adjacent frames with the current frame. This module effectively reduces the computational complexity of feature extraction by capsule networks and improves tracking stability. Experimental evaluations demonstrate that the novel framework achieves superior performance compared to current methods, attaining a 70.3% success rate and 88.4% accuracy on near-infrared (NIR) data. Meanwhile, for visible spectrum (VIS) data analysis, the architecture maintains competitive effectiveness with a 59.6% success rate and 78.8% precision.