In the field of hyperspectral image classification, using spatial information as a supplement to spectral information has been widely applied. This paper proposes a novel dual-scale complementary spatial-spectral joint classification model (DSCSM) to mitigate the issues of detail loss and insufficient utilization of spatial information, which traditionally lead to lower classification accuracy. In essence, the final classification result is obtained through decision fusion of two complementary feature extraction stages. In the pre-processing stage, a new dual-scale truncated filtering feature extraction method (DTFE) is proposed, which uses truncated filters with two different parameter settings to obtain two scales of smoothed patches, and then fuses them to obtain dual-scale structural features using Kernel Principal Component Analysis (KPCA). DTFE preserves edge information while smoothing details, effectively removing noise and retaining the dual-scale feature information. In the post-processing stage, a sub-Markov random walk (subRW) based spatial probability optimization method (SPO) is proposed, which models the spatial association of neighboring pixels, retaining complex textures as well as weak edge information to optimize the classification probability. Finally, the decision fusion strategy is employed to integrate the classification probabilities acquired from the aforementioned two stages. Comparative experiments on six different scene datasets with state-of-the-art classification methods validate that even with a small number of samples, DSCSM can achieve excellent object recognition performance. In addition, comprehensive parameter analysis proves the robustness and stability of the proposed method.

Medical report generation, as a cross-modal automatic text generation task, can be highly significant both in research and clinical fields. The core is to generate diagnosis reports in clinical language from medical images. However, several limitations persist, including a lack of global information, inadequate cross-modal fusion capabilities, and high computational demands. To address these issues, we propose cross-modal global feature fusion Transformer (CGFTrans) to extract global information meanwhile reduce computational strain. Firstly, we introduce mesh recurrent network to capture inter-layer information at different levels to address the absence of global features. Then, we design feature fusion decoder and define ‘mid-fusion’ strategy to separately fuse visual and global features with medical report embeddings, which enhances the ability of the cross-modal joint learning. Finally, we integrate shifted window attention into Transformer encoder to alleviate computational pressure and capture pathological information at multiple scales. Extensive experiments conducted on three datasets demonstrate that the proposed method achieves average increments of 2.9%, 1.5%, and 0.7% in terms of the BLEU-1, METEOR and ROUGE-L metrics, respectively. Besides, it achieves average increments −22.4% and 17.3% training time and images throughput, respectively.

Medical images provide a visual representation of the internal structure of the human body. Injecting a contrast agent can increase the contrast of diseased tissues and assist in the accurate identification and assessment of conditions. Considering the adverse reactions and side effects caused by contrast agents, previous methods synthesized post-contrast images with pre-contrast images to bypass the administration process. However, existing methods pay inadequate attention to reasonable mapping of the lesion area and ignore gaps between post-contrast and real images in the frequency domain. Thus, in this paper, we propose an interactive frequency generative adversarial network (IFGAN) to solve the above problems and synthesize post-contrast images from pre-contrast images. We first designed an enhanced interaction module that is embedded in the generator to focus on the contrast enhancement region. Within it, target and reconstruction branch features interact to control the local contrast enhancement region feature and maintain the anatomical structure. We propose focal frequency loss to ensure the consistency of post-contrast and real images in the frequency domain. The experimental results demonstrated that IFGAN outperforms other sophisticated approaches in terms of preserving the accurate contrast enhancement of lesion regions and anatomical structures. Specifically, our method produces substantial improvements of 7.9% in structural similarity (SSIM), 36.3% in the peak signal-to-noise ratio (PSNR), and 8.5% in multiscale structural similarity (MSIM) compared with recent state-of-the-art methods.

With the development of sensor technology, the sources of remotely sensed image data for the same region are becoming increasingly diverse. Unlike single-source remote sensing image data, multisource remote sensing image data can provide complementary information for the same feature, promoting its recognition. The effective utilization of remote sensing image data from various sources can enhance the extraction of image features and improve the accuracy of feature recognition. Hyperspectral remote sensing (HSI) data and light detection and ranging (LiDAR) data can provide complementary information from different perspectives and are frequently combined in feature identification tasks. However, the process of joint use suffers from data redundancy, low classification accuracy and high time complexity. To address the aforementioned issues and improve feature recognition in classification tasks, this paper introduces a multiprobability decision fusion (PRDRMF) method for the combined classification of HSI and LiDAR data. First, the original HSI data and LiDAR data are downscaled via the principal component–relative total variation (PRTV) method to remove redundant information. In the multifeature extraction module, the local texture features and spatial features of the image are extracted to consider the local texture and spatial structure of the image data. This is achieved by utilizing the local binary pattern (LBP) and extended multiattribute profile (EMAP) for the two types of data after dimensionality reduction. The four extracted features are subsequently input into the corresponding kernel–extreme learning machine (KELM), which has a simple structure and good classification performance, to obtain four classification probability matrices (CPMs). Finally, the four CPMs are fused via a multiprobability decision fusion method to obtain the optimal classification results. Comparison experiments on four classical HSI and LiDAR datasets demonstrate that the method proposed in this paper achieves high classification performance while reducing the overall time complexity of the method.

Hashing methods have made significant progress in cross-modal retrieval tasks with fast query speed and low storage cost. Among them, deep learning-based hashing achieves better performance on large-scale data due to its excellent extraction and representation ability for nonlinear heterogeneous features. However, there are still two main challenges in catastrophic forgetting when data with new categories arrive continuously, and time-consuming for non-continuous hashing retrieval to retrain for updating. To this end, we, in this paper, propose a novel deep lifelong cross-modal hashing to achieve lifelong hashing retrieval instead of re-training hash function repeatedly when new data arrive. Specifically, we design lifelong learning strategy to update hash functions by directly training the incremental data instead of retraining new hash functions using all the accumulated data, which significantly reduce training time. Then, we propose lifelong hashing loss to enable original hash codes participate in lifelong learning but remain invariant, and further preserve the similarity and dis-similarity among original and incremental hash codes to maintain performance. Additionally, considering distribution heterogeneity when new data arriving continuously, we introduce enhanced-semantic similarity to supervise hash learning, and it has been proven that the similarity improves performance with detailed analysis. Experimental results on benchmark datasets show that our proposed method achieves comparative performance comparing with recent state-of-the-art cross-modal hashing methods, and it yields substantial average increments over 20% in retrieval accuracy and almost reduces over 80% training time when new data arrives continuously.