Joint Multimodal Entity-Relation Extraction (JMERE) aims to extract entity-relationship triples in texts from given image-text pairs. As a joint multimodal information extraction task, it has attracted increasing research interest. Previous works of JMERE typically utilize graph networks to align textual entities and visual objects and achieve promising performance. However, these methods do not pay attention to the inconsistency between text and image and the straight alignment could limit the performance of JMERE models. In this paper, we propose a Consistency-adaptive text-image Alignment Generation (CAG) framework for various text-image consistency scenarios. Specifically, we propose a Consistency Factor (CF) to measure the consistency between images and texts. We also design consistency-adaptive contrastive learning based on CF, which can reduce the impact of inconsistent visual and textual information. Additionally, we adopt JMERE-specifical instruction tuning for better entity-relationship triplet generation. Experimental results on the JMERE dataset demonstrate that our proposed CAG is effective and achieves state-of-the-art performance.

Convolutional neural networks have achieved great success in analyzing potential features inside tropical cyclones (TCs) using satellite images for intensity estimation. However, due to the high similarity of visual features in TC images, it is still a challenge to learn the accurate mapping between TC images and numerical intensity. Existing works mainly focus on the visual features of a single TC, ignoring the impact of intensity continuity and time evolution among TCs on decision making. Therefore, we propose a DR-transformer framework for temporal TC intensity estimation. Inside DR-transformers, a novel DR-extractor can extract Distance-consistency(DC) and Rotation-invariance (RI) features between TC images, and therefore can better learn the contours, structures, and other visual features of each TC image. DC features can reduce the estimation error between adjacent intensities, and RI features can eliminate feature deviation caused by shooting angles and TC rotation. Additionally, a transformer with a DR-extractor as the backbone is applied to aggregate the temporal correlation in a series of TC images, which can learn the evolution from intensity to the visual features of TC. Experiments show that the final result, an RMSE of 7.76 knots, outperforms the baseline, and is better than any previously reported method trained on the TCIR dataset.

Visual relationship detection (VRD), a challenging task in the image understanding, suffers from vague connection between relationship patterns and visual appearance. This issue is caused by the high diversity of relationship-independent visual appearance, where inexplicit and redundant cues may not contribute to the relationship detection, even confuse the detector. Previous relationship detection models have shown remarkable progress in leveraging external textual information or scene-level interaction to complement relationship detection cues. In this work, we propose Contextual Coefficients Excitation Feature (CCEF), a focal visual representation, which is adaptively recalibrated from original visual feature responses by explicitly modeling the interdependencies between features and their contextual coefficients. Specifically, contextual coefficients are obtained by calculation of both the spatial coefficients and generated-label ones. In addition, a conditional Wasserstein Generative Adversarial Network (WGAN) regularized with a relationship classification loss is designed to alleviate inadequate training of generated-label coefficients due to long tail distribution of relationship. Experimental results demonstrate the effective improvements of our method on relationship detection. In particular, our method improves the recall from 8.5% to 23.2% of predicting unseen relationship from zero-shot set.