Deep learning (DL)-based object detection algorithms have gained impressive achievements in natural images and have gradually matured in recent years. However, compared with natural images, remote sensing images are faced with severe challenges due to the complex backgrounds and difficult detection of small objects in dense scenes. To address these problems, a novel one-stage object detection model named MDCT is proposed based on a multi-kernel dilated convolution (MDC) block and transformer block. Firstly, a new feature enhancement module, MDC block, is developed in the one-stage object detection model to enhance small objects’ ontology and adjacent spatial features. Secondly, we integrate a transformer block into the neck network of the one-stage object detection model in order to prevent the loss of object information in complex backgrounds and dense scenes. Finally, a depthwise separable convolution is introduced to each MDC block to reduce the computational cost. We conduct experiments on three datasets: DIOR, DOTA, and NWPU VHR-10. Compared with the YOLOv5, our model improves the object detection accuracy by 2.3%, 0.9%, and 2.9% on the DIOR, DOTA, and NWPU VHR-10 datasets, respectively.

Image-text matching has become a challenging task in the multimedia analysis field. Many advanced methods have been used to explore local and global cross-modal correspondence in matching. However, most methods ignore the importance of eliminating potential irrelevant features in the original features of each modality and cross-modal common feature. Moreover, the features extracted from regions in images and words in sentences contain cluttered background noise and different occlusion noise, which negatively affects alignment. Different from these methods, we propose a novel DCT-Transformer Adversarial Network (DTAN) for image-text matching in this paper. This work can obtain an effective metric based on two aspects: i) DCT-Transformer uses DCT (Discrete Cosine Transform) method based on a transformer mechanism to extract multi-domain common representations and eliminate irrelevant features from different modalities (inter-modal). Among them, DCT divides multi-modal content into chunks of different frequencies and quantifies them. ii) The adversarial network introduces an adversary idea by combining the original features of various single modalities and the multi-domain common representation, alleviating the background noise within each modality (intra-modal). The proposed adversarial feature augmentation method can easily obtain the common representation that is only useful for alignment. Extensive experiments are completed on the benchmark datasets Flickr30K and MS-COCO, demonstrating the superiority of the DTAN model over the state-of-the-art methods.

The incompleteness of knowledge graphs (KGs) negatively impacts the performance of KGs in downstream applications (e.g., recommendation systems and information retrieval). This phenomenon has brought an increasing rise in research related to knowledge graph reasoning. Recently, emerged reinforcement learning (RL)-based multi-hop reasoning methods can infer missing information through multi-hop reasoning according to the existing information in KGs, which has better reasoning performance and interpretability. However, these methods always use relation-entity pairs that have been pre-cropped as the action space of agents for path reasoning, which leads to two problems: 1) insufficient learning and reasoning ability of reasoning models and 2) the hard convergence of the training process of agents. To address these problems, we propose a H ierarchical S equence M ulti H op (HSMH) reasoning framework, which consists of the interactive search reasoning model, local-global knowledge fusion mechanism, and action optimization mechanism. We use interactive search reasoning models to select relations and entities independently, thus fully mining the semantic information of relations and entities and improving the learning and reasoning ability of reasoning models. In the HSMH framework, we design the local-global knowledge fusion and action optimization mechanisms for path reasoning, which can enhance agents' state information and action space. Specifically, the local-global knowledge fusion mechanism is designed to acquire the local knowledge of entities and neighboring relations and the global knowledge about KG structure. This local-global knowledge can improve the learning ability of reasoning models. In addition, the action optimization mechanism can combine the filtered action space and the additional action space for efficient path reasoning for agents. Experimental results on five benchmark datasets show that our proposed HSMH framework comprehensively outperforms the state-of-the-art multi-hop reasoning model.

This paper targets to learn multiple comprehensive text prompts that can describe the visual concepts from coarse to fine, thereby endowing pre-trained VLMs with better transfer ability to various downstream tasks. We focus on exploring this idea on transformer-based VLMs since this kind of architecture achieves more compelling performances than CNN-based ones. Unfortunately, unlike CNNs, the transformer-based visual encoder of pre-trained VLMs cannot naturally provide discriminative and representative local visual information. To solve this problem, we propose Frequency-based Comprehensive Prompt Learning (FCPrompt) to excavate representative local visual information from the redundant output features of the visual encoder. FCPrompt transforms these features into frequency domain via Discrete Cosine Transform (DCT). Taking the advantages of energy concentration and information orthogonality of DCT, we can obtain compact, informative and disentangled local visual information by leveraging specific frequency components of the transformed frequency features. To better fit with transformer architectures, FCPrompt further adopts and optimizes different text prompts to respectively align with the global and frequency-based local visual information via a dual-branch framework. Finally, the learned text prompts can thus describe the entire visual concepts from coarse to fine comprehensively. Extensive experiments indicate that FCPrompt achieves the state-of-the-art performances on various benchmarks.

Cross-modal retrieval exhibits significant promise within the realm of multimedia analysis. Numerous sophisticated techniques have gained widespread adoption for harnessing attention mechanisms to facilitate cross-modal correspondence in matching tasks. However, most existing methods learn cross-modal attention based on conventional likelihood, leading to the fine-gained matching from regions and words containing numerous invalid local relationships and false global connections. This phenomenon will bring negative effects on alignment. Different from these methods, we propose a novel Causal Matching Learning (CausMatch) with counterfactual preference framework for cross-modal retrieval in this paper. This work seeks to ascertain the matching relation by incorporating a counterfactual causality preference, enhancing the quality of attention and providing a robust supervisory signal for the learning process are pivotal objectives. This study specifically investigates the influence of acquired visual and textual attention on network predictions by employing counterfactual intervention as a method of scrutiny. This approach aims to discern and analyze the consequential effects on the learning process. Our approach involves maximizing the positive influence to incentivize the network to assimilate more pertinent relationships and connections conducive to fine-grained cross-modal retrieval. The proposed CausMatch model’s effectiveness is systematically substantiated through a comprehensive series of experiments conducted on two widely recognized benchmark datasets, namely MS-COCO and Flickr30K. The results unequivocally demonstrate its superiority over existing state-of-the-art methods, underscoring its robust performance in cross-modal retrieval tasks.