RGBT tracking has received extensive attention as a research hotspot, benefiting from the complementarity of visible (RGB) and thermal (T) modalities, which can better handle tracking tasks in harsh environments. Existing studies typically deploy the same structure to extract features of heterogeneous modalities with different properties, which may be difficult to simultaneously search for mining strategies suitable for each modality, resulting in insufficient feature representation. And, with the success of Transformer, how to jointly consider the heterogeneity and global mining to facilitate better integration of heterogeneous modalities is also a crucial issue worth exploring. To solve these problems, we propose a unique differential reinforcement and global collaboration network for RGBT tracking, which includes a parallel backbone, modal differential reinforcement (MDR), and global mining collaboration module (GMCM). Specifically, MDR introduces raw data to control adaptive differential reinforcement for both modalities via a modality-sharing augmentation unit and a modality-aware gating unit. The data-driven design makes it easier for the two modalities to search for the respective appropriate mining strategies, further effectively enhancing the respective feature representations. GMCM first mines intra-modal and cross-modal global information separately, and then performs an adaptive ensemble operation to balance the two. This operation learns intra-modal and cross-modal global weights in RGB and T modalities to better integrate heterogeneous modalities based on heterogeneity and global cues. Extensive experiments on three datasets demonstrate that the performance of the proposed network is comparable to state-of-the-art RGBT trackers.

Medical image fusion technology is an essential component of computer-aided diagnosis, which aims to extract useful cross-modality cues from raw signals to generate high-quality fused images. Many advanced methods focus on designing fusion rules, but there is still room for improvement in cross-modal information extraction. To this end, we propose a novel encoder-decoder architecture with three technical novelties. First, we divide the medical images into two attributes, namely pixel intensity distribution attributes and texture attributes, and thus design two self-reconstruction tasks to mine as many specific features as possible. Second, we propose a hybrid network combining a CNN and a transformer module to model both long-range and short-range dependencies. Moreover, we construct a self-adaptive weight fusion rule that automatically measures salient features. Extensive experiments on a public medical image dataset and other multimodal datasets show that the proposed method achieves satisfactory performance.

RGBT tracking is a practical solution that combines RGB and thermal infrared modes to solve tracking failures in complex environments to achieve all-day and all-weather work, which makes it gradually applied in multifarious fields. The fundamental reason is that it could avoid the damage of tracking performance caused by the limitation of the imaging characteristics of a single sensor. The existing work aggregates features in different ways, without considering hierarchical complementary interactions and the value of the initial input that may affect subsequent aggregation. In this paper, a novel hierarchical dual-sensor interaction network is proposed, which is mainly composed of dual-sensor interaction, sensor-specific and instance learning. Specifically, our network mainly benefits from the design of two modules, called feature interaction module and data encoding module. The dominant information of the dual sensor is extracted and supplemented by the former based on attention. The latter encodes the raw data into the initial input of the first feature interaction module, whose quality has a key influence on the follow-up. We investigate the performance through extensive experiments compared with the recent state-of-the-art RGB and RGBT trackers on the GTOT and RGBT234 datasets, which verify that our network is effective in both quantitative and qualitative evaluation.

RGB and thermal infrared (RGBT) tracking as a solution in complex environments has gradually become a research hotspot. The powerful complementarity between RGB and thermal infrared data enables trackers to work 24/7. Existing works usually adopt the symmetric network structure that deploys the identical strategy to mine modalities with different properties, ignoring the heterogeneity among modalities. In this article, we propose a novel asymmetric global–local mutual integration network via comprehensively considering symmetric structure, heterogeneity-based global association, and interframe communication. It consists of asymmetric mode-distinguishing parallel structure (AMPS), cross-modal global–local interaction, and interframe monitoring strategy (IMS). Specifically, the AMPS performs discriminative mining on the information of the two modalities by combining the discount module and the branch cement module, and extracts multiscale cues through the multiscale auxiliary module to handle the challenges of scale variation and small-size objects. Then, the global mining module is deployed in the cross-modal global–local interaction section to jointly perform intramodal and intermodal global correlation while acting as the global complement to local feature extraction. Finally, the IMS employs a fast optical flow algorithm to detect interframe displacement to assist the network in better handling camera and fast object motion. Extensive experiments on GTOT, RGBT234, and LasHeR datasets adequately verify the effectiveness of the proposed network, and further ablation experiments also confirm the efficacy of the asymmetric structure and components.

Image fusion aims to integrate the significant information from multi-modal images to obtain a single image. In this paper, a fractional-order variation with convolution norm is proposed for infrared and visible image fusion (IVIF). In our model, a convolution norm l _⊗ is developed for infrared and visible fidelity terms, which can provide the structural group sparseness, alleviating the residual offset. Moreover, the fractional-order variation is employed instead of total variation (TV), preserving more details, and avoiding the staircase effect. Simultaneously, a focused term is utilized to prevent the luminance degradation of the fused result. Finally, extensive experiments are performed on public datasets. The results exhibit that the proposed method achieves better performance in subjective and objective evaluation compared with the excellent methods.