Automatic text summarization is a fundamental NLP application that aims to condense a source text into a shorter version. The rapid increase in multimedia data transmission over the Internet necessitates multi-modal summarization (MMS) from asynchronous collections of text, image, audio and video. In this work, we propose an extractive MMS method that unites the techniques of NLP, speech processing and computer vision to explore the rich information contained in multi-modal data and to improve the quality of MMS. The key idea is to bridge the semantic gaps between multi-modal content. For audio information, we design an approach to selectively use its transcription and to infer the salience of the transcription with audio signals. For visual information, we learn the joint representations of text and images using a neural network. Then, we capture the coverage of the generated summary for important visual information through text-image matching or multi-modal topic modeling. Finally, all the multi-modal aspects are considered to generate a textual summary by maximizing the salience, non-redundancy, readability and coverage through the budgeted optimization of submodular functions. We further introduce a publicly available MMS corpus in English and Chinese. The experimental results demonstrate that our methods outperform other competitive baseline methods.
13 October 2023 in IEEE/ACM Transactions on Audio, Speech, and Language Processing
Text image machine translation (TIMT) aims at directly translating text in the source language embedded in images into the target language. Most existing systems follow the cascaded pipeline diagram from recognition to translation, which suffers from the problem of error propagation, parameter redundancy, and information reduction. The end-to-end model has the potential to alleviate these issues via bridging the recognition and translation models. However, the challenge is the data limitation and modality gap between text and image. In this paper, we propose a novel end-to-end model, namely Modal contrastive learning based End-to-end Text Image Machine Translation (METIMT), which alleviates these issues through end-to-end text image machine translation architecture and modal contrastive learning. Specifically, an image encoder is designed to encode images into the same feature space of corresponding text sentences, with the guidance of an intramodal and inter-modal contrastive learning module. To further promote the research of text image machine translation, we have constructed one synthetic and two real-world datasets. Extensive experiments show that our lighter, faster model outperforms not only existing pipeline methods but also state-of-the-art end-to-end models on both synthetic and real-world evaluation sets. Our code and dataset will be released to the public.
17 January 2025 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Document Image Translation (DIT) aims to translate texts on document images from one language to another. It is a multi-modal task involving cooperation of text and layout. Current approaches either handle layout and translation as separate processes, risking accumulative errors, or use vanilla end-to-end encoder-decoder models to capture layout implicitly, often suffering inadequate layout incorporation. We argue that a favorable framework should explicitly engage layout-specific modules and properly organize them toward translation. For this, we first revisit two key layouts: the geometric layout reflecting word's spatial positions, and the logical layout depicting word's logical order. Then, a novel pipeline (understand layout $\rightarrow$ translate text) is determined to prioritize layouts such that preceding layouts contribute to translation. Following this pipeline, we introduce Unified Document Image Translation (UniDIT), a comprehensive framework that unifies layout with translation in one network. It is devised to leverage each module's advantage, and provide an elaborate feature-conductive flow for module communication globally. A novel bridging mechanism is also introduced to adapt layout features conducive to translation. We further contribute DITransv2, a large-scale fine-grained benchmark that includes heterogeneous and complex document layouts. Extensive experiments on DITransv2 and additional established benchmarks demonstrate UniDIT outperforms previous state-of-the-arts in all aspects.
Document Image Translation (DIT) aims to translate documents in images from one language to another. It is a multi-modal task that involves the cooperation of text, visual layout, and reading logical order. However, existing text-based or vision-based methods rely on solely textual or visual features. Layout-based methods are multi-modal but largely overlook the crucial reading logical order. To fully leverage the multi-modal information and exploit explicit modules to learn better reading logical order for DIT, this paper proposes the “reading-when-translating” guideline. It collaborates the translation process with an auxiliary “reading” process such that reading logical order directly contributes to translation. Following this guideline, we propose Document Reading and Translation Network (DocRTN), a novel unified framework that seamlessly integrates reading order into DIT, enabling the model to handle complex layouts by “reading” text in a human-like, coherent sequence. The unified framework comprises a reading flow decoder and a translation decoder. A novel feature decorator is proposed to harmonize the reading and translation channels, ensuring reading features are optimally adapted for translation. Extensive comparisons and analysis on 5 domains and 4 translation directions demonstrate DocRTN outperforms previous state-of-the-arts in all aspects.