The demand for edge device models equipped with multilingual visual capabilities is rapidly increasing in complex IoT application scenarios. While many studies have endowed models with strong visual sensory and language analysis capabilities, these models are often large and require substantial amounts of data. Moreover, multilingual parallel corpora are extremely scarce. Although large parameter sizes can enhance a model’s visual-language processing capabilities, the high training and inference costs make them unsuitable for edge devices and result in suboptimal performance in multilingual contexts. To address these challenges, this article proposes an generative visual language model, that is, cross-lingual, lightweight, data-efficient, easy to train, and easy to infer. We map both English and non-English features into the same space and align them with a visually distilled model while leveraging the inherent similarity information of languages to increase the supervision coverage of the dataset. Through extensive experiments, we demonstrate that our model achieves state-of-the-art performance across three downstream tasks: 1) image captioning; 2) machine translation; and 3) visual question answering, surpassing existing methods.

Data modeling is a critical component in the development of next-generation consumer electronics (CE), as it provides massive data support to implement smart services related to CE products, such as complex correlation analysis of the CE market, the development of technology, and consumer behaviors. However, the existing works still face the following challenges: (1) the manual data processing is time-consuming and labor-intensive while analyzing the massive data with domain knowledge; (2) insufficient attention is given to the analysis of relationships among data collections; and (3) redundant data is not eliminated, leading to excess computational burden. To address these challenges, we propose an efficient automatic data modeling method that employs domain-aware knowledge, significantly reducing the cost of data modeling. Our approach starts by leveraging a knowledge-based classifier to extract domain-related resources from open common single-document summarization datasets (OC-SDS), thus reducing data acquisition expenses. Then after collecting related documents by original summaries, a two-stage filtering process is applied to eliminate redundant and non-related documents. Finally, the original summaries are iteratively updated until reaching the threshold, enhancing informativeness and introducing novel expressions without the need for manual marking. As a practical application, we take emergency news as an example and built a typical dataset based on our method. After an extensive analysis, the empirical results evidence that our method is superior in scale and quality compared to the existing methods, with over 13.7% more related data, providing a valuable contribution to the field of CE product development.

Aspect-level sentiment analysis within multimodal contexts, focusing on the precise identification and interpretation of sentiment attitudes linked to the target aspect across diverse data modalities, remains a focal research area that perpetuates the advancement of discourse and innovation in artificial intelligence. However, most existing methods tend to focus on extracting visual features from only one facet, such as face expression, which ignores the value of information from other key facets, such as the textual information presented by the image modality, resulting in information loss. To overcome the aforementioned constraint, we put forth a novel approach designated as Multi-faceted Information Extraction and Cross-mixture Fusion (MIECF) for Multimodal Aspect-based Sentiment Analysis. Our approach captures more comprehensive visual information in the image and integrates these local and global key features from multiple facets. Local features, such as facial expressions and textual features, provide direct and rich emotional cues. By contrast, the global feature often reflects the overall emotional atmosphere and context. To enhance the visual representation, we designed a Cross-mixture Fusion method to integrate this local and global multimodal information. In particular, the method establishes semantic relationships between local and global features to eliminate ambiguity brought by single-facet information and achieve more accurate contextual understanding, providing a richer and more precise manner for sentiment analysis. The experimental findings indicate that our proposed approach achieves a leading level of performance, resulting in an Accuracy of 79.65 % on the Twitter-2015 dataset, and Macro-F1 scores of 75.90 % and 73.11 % for the Twitter-2015 and Twitter-2017 datasets, respectively.

As multilingual artificial intelligence systems proliferate, achieving robust cross-lingual understanding remains an open challenge. Recent works have made progress on visual question answering (VQA) models by pretraining on large English image-text datasets. However, there is a language gap as most models are English-centric. Existing attempts at multilingual VQA rely on machine translation or multilingual model pretraining, but cannot effectively transfer rich cross-modal knowledge from English models. In this work, we propose the cross-lingual multimodal knowledge transfer (CMKT) framework to efficiently extend English VQA models to non-English languages via knowledge distillation. Specifically, we introduce a code-mixed cross-lingual mask modeling (CCM) method to establish representations for new languages using small image-text data. We also design a multimodal knowledge distillation (MMKD) method to transfer modal understanding from English models by imitating their sequence processing. Experiments on the xGQA benchmark demonstrate that CMKT can effectively improve zero-shot learning and few-shot learning in non-English languages. Our method reduces the data and computation needed to train multilingual VQA models from scratch. The knowledge transfer paradigm enables non-English languages to inherit and generalize the intricate visual-semantic relationships learned from English. The results also show that the proposed method outperforms previous state-of-the-art methods in the zero-shot setting on the xGQA dataset.

Multi-modal knowledge graphs (MMKGs) enrich the semantic expression capabilities of traditional knowledge graphs by incorporating diverse modal information, showcasing immense potential in various knowledge reasoning tasks. However, existing MMKGs encounter numerous challenges in the link prediction task (i.e., knowledge graph completion reasoning), primarily due to the complexity and diversity of modal information and the imbalance in quality. These challenges make the efficient fusion and enhancement of multi-modal information difficult to achieve. Most existing methods adopt simple concatenation or weighted fusion of modal features, but such approaches fail to fully capture the deep semantic interactions between modalities and perform poorly when confronted with modal noise or missing information. To address these issues, this paper proposes a novel framework model-Adaptive Fusion and Modality Information Enhancement(AFME). This framework consists of two parts: the Modal Information Fusion module (MoIFu) and the Modal Information Enhancement module (MoIEn). By introducing a relationship-driven denoising mechanism and a dynamic weight allocation mechanism, the framework achieves efficient adaptive fusion of multi-modal information. It employs a generative adversarial network (GAN) structure to enable global guidance of structural modalities over feature modalities and adopts a multi-layer self-attention mechanism to optimize both intra- and inter-modal features. Finally, it jointly optimizes the losses of the triple prediction task and the adversarial generation task. Experimental results demonstrate that the AFME framework significantly improves multi-modal feature utilization and knowledge reasoning capabilities on multiple benchmark datasets, validating its efficiency and superiority in complex multi-modal scenarios.

Large Multimodal Models (LMMs) excel in English multimedia tasks but face challenges in adapting to other languages due to linguistic diversity, limited non-English multimodal data, and high training costs. Existing approaches rely on machine-translated multimodal corpora or multilingual large language models, yet they demand substantial resources and achieve only modest zero-shot cross-lingual transfer performance, as shown in the IGLUE benchmark. In this work, we propose SMSA, a Syntax-aware Multimodal Semantic Adaptation approach, which efficiently extends vision-language models (VLMs) to multiple languages via a lightweight adaptation module. Instead of learning from scratch, SMSA transfers multimodal knowledge from English-trained models using two key components: (1) a Syntax-aware Adapter (SAA), which restructures multilingual text representations to align better with English syntax, reducing cross-lingual misalignment; (2) a Multimodal Semantic Distillation (MSD) method, which enables the model to mimic English sequence processing and retain multimodal associations across languages. This allows efficient adaptation to new languages while preserving the original model's strong multimodal capabilities. We extend an MoE-based VLM to 8 languages using a small translation dataset. Evaluations on the IGLUE benchmark show that SMSA achieves strong zero-shot transfer, outperforming some multilingual LMMs and demonstrating its effectiveness in cross-lingual vision-language adaptation.