Integrating multi-modal data in natural language processing has opened new pathways for the enhancement of dialogue summarization. However, existing models often struggle to effectively synthesize textual, auditory, and visual inputs. This paper introduces a Modality-Enhanced Multi-Channel Attention Network (MEMA), a novel approach designed to optimize the integration and interaction of diverse modalities for dialogue summarization. MEMA leverages symmetrical embedding strategies to balance the integrity and distinctiveness of each modality, ensuring a harmonious interaction within the unified architecture. By maintaining symmetry in the processing flow, MEMA enhances the contextual richness and coherence of the generated summaries. Our model demonstrates superior performance on the Multi-modal Dialogue Summarization (MDS) dataset, particularly in generating contextually enriched abstract summaries. The results underscore MEMA’s potential to transform dialogue summarization by providing a more symmetrical and integrated understanding of multi-modal interactions, bridging the gap in multi-modal data processing, and setting a new standard for future summarization tasks.

Multimodal summarization, a rapidly evolving field within multimodal learning, focuses on generating cohesive summaries by integrating information from diverse modalities, such as text and images. Unlike traditional unimodal summarization, multimodal summarization presents unique challenges, particularly in capturing fine-grained interactions between modalities. Current models often fail to account for complex cross-modal interactions, leading to suboptimal performance and an over-reliance on one modality. To address these issues, we propose a novel framework, hypergraph transformer-based multimodal summarization (HGTMFS), designed to model high-order relationships across modalities. HGTMFS constructs a hypergraph that incorporates both textual and visual nodes and leverages transformer mechanisms to propagate information within the hypergraph. This approach enables the efficient exchange of multimodal data and improves the integration of fine-grained semantic relationships. Experimental results on several benchmark datasets demonstrate that HGTMFS outperforms state-of-the-art methods in multimodal summarization.