Ruiyang Huang

Ruiyang Huang


Research Focus & Metrics

Academic Career Length

5Years (2022-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 June 2023 in Applied Sciences
Verified
Journal Article
Open access
PDF file
HWHuansha Wang
Huansha Wang
QLQinrang Liu
Qinrang Liu
RHRuiyang Huang
Ruiyang Huang
JZJianpeng Zhang
Jianpeng Zhang
Multi-modal entity alignment refers to identifying equivalent entities between two different multi-modal knowledge graphs that consist of multi-modal information such as structural triples and descriptive images. Most previous multi-modal entity alignment methods have mainly used corresponding encoders of each modality to encode entity information and then perform feature fusion to obtain the multi-modal joint representation. However, this approach does not fully utilize the multi-modal information of aligned entities. To address this issue, we propose MEAFE, a multi-modal entity alignment method based on feature enhancement. The MEAFE adopts the multi-modal pre-trained model, OCR model, and GATv2 network to enhance the model’s ability to extract useful features in entity structure triplet information and image description, respectively, thereby generating more effective multi-modal representations. Secondly, it further adds modal distribution information of the entity to enhance the model’s understanding and modeling ability of the multi-modal information. Experiments on bilingual and cross-graph multi-modal datasets demonstrate that the proposed method outperforms models that use traditional feature extraction methods.

31 October 2022 in Electronics
Verified
Journal Article
Open access
PDF file
HWHuansha Wang
Huansha Wang
RHRuiyang Huang
Ruiyang Huang
JZJianpeng Zhang
Jianpeng Zhang
Because the pretraining model is not limited by the scale of data annotation and can learn general semantic information, it performs well in tasks related to natural language processing and computer vision. In recent years, more and more attention has been paid to research on the multimodal pretraining model. Many vision–language multimodal datasets and related models have been proposed one after another. In order to better summarize and analyze the development status and future trend of vision–language multimodal pretraining model technology, firstly this paper comprehensively combs the category system and related tasks of vision–language multimodal pretraining. Secondly, research progress on vision–language multimodal pretraining is summarized and analyzed from the two dimensions of image–language and video–language models. Finally, problems with and development trends in vision–language multimodal pretraining are discussed.

1 January 2025 in Computers, Materials & Continua
Verified
Journal Article
Open access
HWHuansha Wang
Huansha Wang
RHRuiyang Huang
Ruiyang Huang
QLQinrang Liu
Qinrang Liu
SLShaomei Li
Shaomei Li
JZJianpeng Zhang
Jianpeng Zhang
Multi-modal knowledge graph completion (MMKGC) aims to complete missing entities or relations in multi-modal knowledge graphs, thereby discovering more previously unknown triples. Due to the continuous growth of data and knowledge and the limitations of data sources, the visual knowledge within the knowledge graphs is generally of low quality, and some entities suffer from the issue of missing visual modality. Nevertheless, previous studies of MMKGC have primarily focused on how to facilitate modality interaction and fusion while neglecting the problems of low modality quality and modality missing. In this case, mainstream MMKGC models only use pre-trained visual encoders to extract features and transfer the semantic information to the joint embeddings through modal fusion, which inevitably suffers from problems such as error propagation and increased uncertainty. To address these problems, we propose a Multi-modal knowledge graph Completion model based on Super-resolution and Detailed Description Generation (MMCSD). Specifically, we leverage a pre-trained residual network to enhance the resolution and improve the quality of the visual modality. Moreover, we design multi-level visual semantic extraction and entity description generation, thereby further extracting entity semantics from structural triples and visual images. Meanwhile, we train a variational multi-modal auto-encoder and utilize a pre-trained multi-modal language model to complement the missing visual features. We conducted experiments on FB15K-237 and DB13K, and the results showed that MMCSD can effectively perform MMKGC and achieve state-of-the-art performance.

14 February 2024 in Applied Sciences
Verified
Journal Article
Open access
PDF file
YPYonghao Pan
Yonghao Pan
HYHongtao Yu
Hongtao Yu
SLShaomei Li
Shaomei Li
RHRuiyang Huang
Ruiyang Huang
In the current deep learning model, the computation between each feature and parameter is defined in the real number field. This, together with the nonlinearity of the deep learning model, makes it difficult to analyze the relationship between the values of the computational process and the original features from computation in the real number field. We extend the operational rules of the deep learning model in space and propose the orthogonal neural network (ONN) model, in which the features are set orthogonally to each other in space by “modulating” each input feature of the deep learning model to different orthogonal bases. Because the modulated numerical features are orthogonal to each other, they can be separated from the computations of the ONN model. By “demodulating” the model during and after the calculation, we can obtain a numerical relationship between the results and the original features, which can further provide theoretical and computational support for our analysis of the model. Finally, we compute the weights for each input feature as an interpretable deep learning approach, and describe how the model focuses attention on each feature based on the application of the orthogonal neural network model on two typical models: convolutional neural networks and graph neural networks.

1 October 2022 in Electronics
Verified
Journal Article
Open access
PDF file
HWHuansha Wang
Huansha Wang
RHRuiyang Huang
Ruiyang Huang
JZJianpeng Zhang
Jianpeng Zhang
Entity alignment is used to determine whether entities from different sources refer to the same object in the real world. It is one of the key technologies for constructing large-scale knowledge graphs and is widely used in the fields of knowledge graphs and knowledge complementation. Because of the lack of semantic connection between the visual modality face attribute of the person entity and the text modality attribute and relationship information, it is difficult to model the visual and text modality into the same semantic space, and, as a result, that the traditional multimodal entity alignment method cannot be applied. In view of the scarcity of multimodal person relation graphs datasets and the difficulty of the multimodal semantic modeling of person entities, this paper analyzes and crawls open-source semi-structured data from different sources to build a multimodal person entity alignment dataset and focuses on using the facial and semantic information of multimodal person entities to improve the similarity of entity structural features which are modeled using the graph convolution layer and the dynamic graph attention layer to calculate the similarity. Through verification on the self-made multimodal person entity alignment dataset, the method proposed in this paper is compared with other entity alignment models which have a similar structure. Compared with AliNet, the probability that the first item in the candidate pre-aligned entity set is correct is increased by 12.4% and average ranking of correctly aligned entities in the candidate pre-aligned entity set decreased by 32.8, which proves the positive effect of integrating multimodal facial information, applying dynamic graph attention and a layer-wise gated network to improve the alignment effect of person entities.

1 January 2025 in Computers, Materials & Continua
Verified
Journal Article
Open access
PDF file
HWHuansha Wang
Huansha Wang
RHRuiyang Huang
Ruiyang Huang
QLQinrang Liu
Qinrang Liu
XWXinghao Wang
Xinghao Wang
Multi-modal Named Entity Recognition (MNER) aims to better identify meaningful textual entities by integrating information from images. Previous work has focused on extracting visual semantics at a fine-grained level, or obta... | Find, read and cite all the research you need on Tech Science Press