5 October 2022 in IEEE Transactions on Geoscience and Remote Sensing
The synthetic aperture radar (SAR) system is a kind of active remote sensing, which can be carried on a variety of flight platforms and can observe the Earth under all-day and all-weather conditions, so it has a wide range of applications. However, the interpretation of SAR images is quite challenging and not suitable for nonexperts. In order to enhance the visual effect of SAR images, this article proposes a multiscale generative adversarial network based on wavelet feature learning (WFLM-GAN) to implement the translation from SAR images to optical images; the translated images not only retain the key content of SAR images but also have the style of optical images. The main advantages of this method over the previous SAR-to-optical image translation (S2OIT) methods are given as follows. First, the generator does not learn the mapping from SAR images to optical images directly but learns the mapping from SAR images to wavelet features and then reconstructs the gray-scale images to optimize the content, increasing the mapping relationships and helping to learn more effective features. Second, a multiscale coloring network based on detail learning and style learning is designed to further translate the gray-scale images into optical images, which makes the generated images have an excellent visual effect with details closer to real images. Extensive experiments on SAR image datasets in different regions and seasons demonstrate the superior performance of WFLM-GAN over the baseline algorithms in terms of structural similarity (SSIM), the peak signal-to-noise ratio (PSNR), the Frechet inception distance (FID), and the kernel inception distance (KID). Comprehensive ablation studies are also carried out to isolate the validity of each proposed component. Our codes will be available at https://github.com/G2022G/WFLM-GAN .
25 December 2024 in IEEE Transactions on Multimedia
Image-text retrieval has made remarkable achievements through the development of feature extraction networks and model architectures. However, almost all region featurebased methods face two serious problems when modeling modality interactions. First, region features are prone to feature entanglement in the feature extraction stage, making it difficult to accurately reason complex intra-model relations between visual objects. Second, region features lack rich contextual information, background, and object details, making it difficult to achieve precise inter-modal alignment with textual information. In this paper, we propose a novel Dual Stream Relation Learning Network (DSRLN) to jointly solve these issues with two key components: a Geometry-sensitive Interactive Self-Attention (GISA) module and a Dual Information Fusion (DIF) module. Specifically, GISA extends the vanilla self-attention network from two aspects to better model the intrinsic relationships between different regions, thereby improving high-level visual-semantic reasoning ability. DIF uses grid features as an additional visual information source, and achieves deeper and complex fusion between the two types of features through a masked crossattention module and an adaptive gate fusion module, which can capture comprehensive visual information to learn more precise inter-modal alignment. Besides, our method also learns a more comprehensive hierarchical correspondence between images and sentences through local and global alignment. Experimental results on two public datasets, i.e., Flickr30K and MS-COCO, fully demonstrate the superiority and effectiveness of our model.
15 September 2023 in IEEE Transactions on Multimedia
Current image-text retrieval methods mainly utilize region features that provide object-level information to represent images, making the retrieval results more accurate and interpretable. However, there are several issues with region features, such as lack of rich contextual information, loss of object details and risk of detection redundancy. The ideal visual features in image-text retrieval should have three characteristics: object-level, semantically-rich, and language-aligned. To this end, we propose a novel visual representation framework to capture more comprehensive and powerful visual features. Specifically, since these region feature disadvantages are the grid feature advantages, we first build a two-step interaction model to explore the complex relationship between them from the spatial and semantic perspectives to integrate their complementary information, making the fused visual features both object-level and semantic-rich. Then, we design a text-integrated visual embedding module that utilizes textual information as guidance to filter redundant regions, further endowing visual features with language-aligned capabilities. Finally, we develop a multi-attention pooling module to better aggregate these enhanced visual features in a more fine-grained manner. Extensive experiments demonstrate that our proposed model achieves state-of-the-art performance on the benchmark datasets Flickr30K and MS-COCO.
2 January 2026 in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
Synthetic Aperture Radar (SAR), an active remote sensing technology that can be deployed on various aerial plat forms, generates images by emitting microwaves and analyzing the intensity of backscattered signals. The penetration capability of microwaves enables observation of Earth at all weather, day and night, broadening the application of SAR. However, SAR image interpretation remains challenging for non-experts. In contrast, optical images provide intuitive visual features but are weather-sensitive. To synergize these modalities, SAR-to-Optical Translation (S2OT) has gained research attention, but most studies prioritize visual quality or similarity metrics over practical applicability to downstream tasks. This study proposes a high level vision task-coordinated S2OT framework to address this gap. Semantic segmentation, emulating land cover classification, is integrated as the downstream task. Semantic segmentation loss guides the network to generate optical images that enhance task-relevant features. To preserve structural information in SAR images, we introduce SSIM loss and incorporate SAR derived semantic segmentation maps as auxiliary inputs. An identity loss further aligns the distributions of generated and real optical images, mitigating domain discrepancies. Extensive experiments confirm that S2OT improves downstream land cover classification. The inclusion of task-specific losses elevates translation quality: our HVTC-GAN surpasses the baseline methods in SSIM and PSNR metrics. Ablation studies validate the effectiveness of co-training S2OT with high-level vision tasks, demonstrating that task-oriented constraints enhance both translation fidelity and downstream utility. Code will be available at https://github.com/NWPU-LHH/HVTC-GAN