Image captioning aims at understanding various semantic concepts (e.g., objects and relationships) from an image and integrating them in a sentence-level description. Hence, it is necessary to learn the interaction among these concepts. If we define the context of the interaction to be involved in the subject-predicate-object triplet, most current methods only focus on the single triplet for the first-order interaction to generate sentences. Intuitively, we humans are able to perceive the high-order interaction among concepts from two or more triplets to describe an image. For example, when we see the triplets man-cutting-sandwich and man-with-knife , it is natural to integrate and predict the sentence man cutting sandwich with knife . This depends on the high-order interaction between cutting and knife in different triplets. Therefore, exploiting high-order interaction is expected to benefit image captioning and focus on reasoning. In this paper, we introduce the novel high-order interaction learning method over detected objects and relationships for image captioning under the umbrella of the encoder-decoder framework. We first extract a set of object and relationship features in an image. During the encoding stage, the interactive refining network is proposed to learn high-order representations by modeling intra- and inter-object feature interaction in the self-attention fashion. During the decoding stage, the interactive fusion network is proposed to integrate object and relationship information by strengthening their high-order interaction based on language context for sentence generation. In this way, we learn the object-relationship dependencies in different stages, which can provide abundant cues for both visual understanding and caption generation. Extensive experiments show that the proposed method can achieve competitive performances against the state-of-the-art methods on MSCOCO dataset. Additional ablation studies further validate its effectiveness.

Visual relationship detection aims to describe the interactions between pairs of objects, such as person-ride-bike and bike-next to-car triplets. In reality, it is often the case that there exist some groups of strongly correlated relationships, while others are weakly related. Intuitively, the common relationships can be roughly categorized into three types: geometric (e.g., next to), possessive (e.g., has), or action (e.g., ride). However, previous studies ignore the relatedness discovery among multiple relationships, while only lie on an unified space to leverage visual features or statistical dependencies into categories. To tackle this problem, we propose an adaptively clustering-driven network for visual relationship detection. First, we propose two novel modules to jointly discover the common distribution space and preserve the characteristics of individual relationships for clustering. Then, a fused inference is designed to integrate the group-induced information and the language prior to facilitate the predicate inference. Especially, we design the Frobenius-norm regularization to boost the discriminative clustering. To the best of our knowledge, the proposed method is the first supervised framework to realize subject-predicate-object relationship-aware clustering for visual relationship detection. Extensive experiments show that the proposed method can achieve competing performances against the state-of-the-art methods on the Visual Genome dataset. Additional ablation studies further validate its effectiveness.

In this work, we address the challenge of stable and robust time-varying direction-of-arrival (DOA) tracking, where both the DOA trajectory and the number of sources change over time across different scenarios. We formulate the problem as a multi-label classification task and propose a novel solution leveraging Riemannian manifold geometry and advanced neural networks for sequential data processing. Specifically, we introduce the Complex Hermitian Positive Definite embedding Transformer (C-HPD-T) network, which learns statistical representations of covariance matrices. The C-HPD-T consists of an HPD module and a Transformer module. The HPD module learns an appropriate statistical representation and the complex Transformer module models the data temporally. In particular, in the C-HPD-T network, we build a complex singular value decomposition (SVD) module to ensure that complex structure is preserved during the calculation of each layer. Meanwhile, we design a Transformer-based sequential network to capture the temporal relationship of signals and for the time-varying DOAs. Through training the network with varying numbers of moving targets featuring diverse motion trajectories, the network demonstrates excellent performance and robustness against array imperfections. Extensive simulation results demonstrate its superior performance gains compared to existing methods.

News captioning aims to describe an image with its news article body as input. It greatly relies on a set of detected named entities, including real-world people, organizations, and places. This article exploits commonsense knowledge to understand named entities for news captioning. By “understand,” we mean correlating the news content with commonsense in the wild, which helps an agent to (1) distinguish semantically similar named entities and (2) describe named entities using words outside of training corpora. Our approach consists of three modules: (a) Filter Module aims to clarify the commonsense concerning a named entity from two aspects: what does it mean ? and what is it related to ?, which divide the commonsense into explanatory knowledge and relevant knowledge , respectively. (b) Distinguish Module aggregates explanatory knowledge from node-degree , dependency , and distinguish three aspects to distinguish semantically similar named entities. (c) Enrich Module attaches relevant knowledge to named entities to enrich the entity description by commonsense information (e.g., identity and social position). Finally, all of information is integrated into the large multimodal model to generate the news caption. Extensive experiments on two challenging datasets (i.e., GoodNews and NYTimes) demonstrate the superiority of our method. Ablation studies and visualization further validate its effectiveness in understanding named entities.