Object tracking based on visible images may fail when the visible images are unreliable, for example when the illumination condition is poor. Infrared images reveal thermal radiation of objects and are insensitive to these factors. Due to the complementary features of visible and infrared images, RGBinfrared fusion tracking has attracted widespread attention recently. In this paper, an RGB-infrared fusion tracking method based on the fully convolutional Siamese Networks, termed as SiamFT, is proposed. Visible and infrared images are firstly processed by two Siamese Networks, namely visible network and infrared network, respectively. Then, convolutional features of visible and infrared template images extracted from two Siamese Networks are concatenated to form fused template image. Convolutional features of visible and infrared search image are fused through the proposed feature fusion network adaptively. In particular, a modality weight computation method based on the response value of Siamese network is proposed to predict the reliability of different images. Cross-relation is then applied to the fused template feature and the fused search feature to produce the final response map, based on which the tracking results can be obtained. Extensive experiments indicate that the proposed SiamFT shows better performance than thestate-of-art fusion tracking algorithms at real-time speed.
22 June 2023 in IEEE Transactions on Circuits and Systems for Video Technology
With the increasing diversity of visual tracking tasks, object tracking in RGB and thermal (RGB-T) modalities has received widespread interest. Most of the existing RGB-T tracking methods mainly improve tracking performance by integrating hierarchically complementary information from RGB and thermal modalities, however, they are insufficient in handling tracking failures due to the lack of re-detection capability. To address these issues, we propose a new RGB-T tracking method with online learning samples and adaptive object recovery. First, the features of RGB and thermal modalities are concatenated for robust appearance modeling. Second, a multimodal fusion strategy is designed to stably integrate reliable information of modalities and propose to use similarity to measure tracking confidence. Finally, a detector with online learning of positive and negative samples and adaptive recovery is developed to correct unreliable tracking results. Numerical results on five recent large-scale benchmark datasets demonstrate that the proposed tracker achieves competitive performance compared to other state-of-the-art methods.
The traditional neural network Intelligent chip has the problem of high power consumption due to classical computing architecture, limiting the development of neural network Intelligent chips. Stochastic computing (SC) encodes binary numbers into stochastic pulse sequences in operation, taking advantage of low power consumption and high performance. The application of SC in spiking neural networks (SNNs) Intelligent chips is beneficial to solving the high power consumption of traditional neural network chips. This article first summarizes the basic elements of SNNs and the basic principles of SC. Then, we review the development trends of the stochastic computation-based neural network chips and existing SNN chips under research at home and abroad, respectively, and analyze the current problems. Finally, a review of SNN chips based on SC is highlighted. This paper aims to provide new research directions and to learn ideas for the field of SNN chips through systematic summaries.
Vision–language tracking presents a crucial challenge in multimodal object tracking. Integrating language features and visual features can enhance target localization and improve the stability and accuracy of the tracking process. However, most existing fusion models in vision–language trackers simply concatenate visual and linguistic features without considering their semantic relationships. Such methods fail to distinguish the target’s appearance features from the background, particularly when the target changes dramatically. To address these limitations, we introduce an innovative technique known as multimodal features alignment (MFA) for vision–language tracking. In contrast to basic concatenation methods, our approach employs a factorized bilinear pooling method that conducts squeezing and expanding operations to create a unified feature representation from visual and linguistic features. Moreover, we integrate the co-attention mechanism twice to derive varied weights for the search region, ensuring that higher weights are placed on the aligned visual and linguistic features. Subsequently, the fused feature map with diverse distributed weights serves as the search region during the tracking phase, facilitating anchor-free grounding to predict the target’s location. Extensive experiments are conducted on multiple public datasets, and our proposed tracker obtains a success score of 0.654/0.553/0.447 and a precision score of 0.872/0.556/0.513 on OTB-LANG/LaSOT/TNL2K. These results are satisfying compared with those of recent state-of-the-art vision–language trackers.
9 January 2024 in IEEE Transactions on Instrumentation and Measurement
Different from most existing algorithms that explore the integration of information from RGB and thermal (RGB-T) hierarchical features, we propose a novel adaptive learning of modal information from the decision-level perspective to achieve efficient and robust tracking. In our paradigm, the relative reliability between different modalities is mined by maximizing the peak-to-sidelobe ratio (PSR) model. Synchronously, the learned reliability can also be used to guide the correct update of the target template for each modality. Experiments on widely used large-scale benchmarks demonstrate that our method achieves competitive performance against other state-of-the-art trackers while enabling real-time tracking. Our codes will be available at: https://github.com/Liujunzx/MPT .
1 May 2023 in IEEE Transactions on Instrumentation and Measurement
Jet fuel leaks not only waste resources and increase costs but also pose a risk of emergency landings and aviation accidents. With the blossom and implementation of deep learning, crack segmentation techniques have been rapidly developed in many fields. However, it is struggle to get accurate and complete crack segmentation results because of images’ complex background environment. To address this issue, we collected and labeled a dataset of 2824 crack images from the surface of aircraft fuel tank, named CAFT2800. And this article presents an atrous spatial pyramid fusion and hybrid attention network based on deep learning to deal with the complicated environment. The backbone of network uses a hierarchical structure Swin Transformer to extract features. In the neck of the network, an atrous spatial pyramid fusion module is proposed to further capture contextual information in multiple scale. Unlike the ASPP module, which aggregates image-level information with pooling, we abandon this operation according to the characteristics of crack. And the neck structure is designed for fusing high-level semantic information to all scales. At the gate between the neck and the decoder head, a SK (selective kernel) block is embedded into the network to recalibrates channel-wise feature responses. Due to the morphological characteristics of crack, we propose an evaluation index, Thinning F1 score (TFscore), which is more meaningful compared to the commonly used F1 score. Sufficient control experiments were conducted on the CAFT2800 dataset and two complicated environment benchmarks (DeepCrack and GAPs) to test the effectiveness of the network, and our method achieved superior performance. Source code and the CAFT2800 are available at https://github.com/Gu-EH/CAFT2800.