Ruochen Li

Ruochen Li

Auto-generated profile, based on publicly available information

Ruochen Li

Shandong University of Science and Technology

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
5 February 2024 in IEEE Transactions on Multimedia
Verified
Journal Article
Closed access
MZMinglu Zhao
Minglu Zhao
WWWenmin Wang
Wenmin Wang
TCTongbao Chen
Tongbao Chen
RZRui Zhang
Rui Zhang
RLRuochen Li
Ruochen Li
Recent conditional and unconditional video generation tasks have been accomplished mainly based on generative adversarial network (GAN), diffusion, and autoregressive models. However, in some circumstances, using only one modality cannot provide enough semantic information. Therefore, in this paper, we propose text-audio to video (TA2V) generation, a new task for generating realistic videos from two different guided modalities, text and audio, which has not been explored much thus far. Compared to image generation, video generation is a harder task because of the complexity of processing higher-dimensional data and scarcer suitable datasets, especially for multimodal video generation. To overcome these limitations, (i) we propose the Text&Audio-guided-Video-Maker (TAgVM) model, which consists of two modules: a text-guided video generator and a text&audio-guided video modifier. (ii) This model uses a 3D VQ-GAN to compress high-dimension video data to a low-dimension discrete sequence, followed by an autoregressive model to guide text-conditional generation in the latent space. Then, we apply a text&audio-guided diffusion model to the generated video scenes, providing additional semantic details corresponding to the audio and text. (iii) We introduce a newly produced music performance video dataset, the University of Rochester Multimodal Music Performance with Video-Audio-Text (URMP-VAT), and a landscape dataset, Landscape with Video-Audio-Text (Landscape-VAT), both of which include three modalities (text, audio, and video) that are aligned with each other. The results demonstrate that our model can create videos with satisfactory quality and semantic information. The source code and datasets are available at https://github.com/Minglu58/TA2V.

15 September 2023 in IEEE Transactions on Multimedia
Verified
Journal Article
Closed access
TCTongbao Chen
Tongbao Chen
WWWenmin Wang
Wenmin Wang
ZJZhe Jiang
Zhe Jiang
RLRuochen Li
Ruochen Li
Bingshu WangBingshu Wang
Bingshu Wang
Video corpus moment retrieval has become a hot topic recently, which aims to localize a consequent video moments highly relevant to the given query language description from video corpus. Existing methods towards this challenging task are suffering from the cases when the visual information and textual information in the video are very different from each other or from the cases where the redundant video content is semantically irrelevant with the query language description, which make the model confused of figuring out the truly useful within- and cross-modality information. In this paper, we propose a novel Cross-Modality Knowledge Calibration Network (CKCN) to solve the issue mentioned above. Specifically, a dual calibration transformer module with improved multi-head attention is proposed to simultaneously capture the within- and cross-modality features between the visual and textual modality of the video automatically compressing the redundant information, and then a query-dependent fusion module is designed to guide feature fusion of the video's multi-modal information using the prior knowledge of query which further refine more important modality features. At last, a query-guided calibration transformer module with a well-designed learnable cell is utilized to align the query and video, forming a single joint representation for moment localization. Meanwhile, we introduce transfer learning into the task of video corpus moment retrieval (VCMR) for the first time to solve the defect of insufficient labeled data. Extensive experiments have been conducted on both the widely used TVR dataset and DiDeMo dataset which have achieved new state-of-the-art, thus verifying the effectiveness of our proposed CKCN.

11 June 2024 in Neurocomputing
Verified
Journal Article
Closed access
TCTongbao Chen
Tongbao Chen
WWWenmin Wang
Wenmin Wang
MZMinglu Zhao
Minglu Zhao
RLRuochen Li
Ruochen Li
ZJZhe Jiang
Zhe Jiang
CYCheng Yu
Cheng Yu

4 April 2024 in Neurocomputing
Verified
Journal Article
Closed access
RLRuochen Li
Ruochen Li
WWWenmin Wang
Wenmin Wang
HHHuanqiang Hu
Huanqiang Hu
TCTongbao Chen
Tongbao Chen
MZMinglu Zhao
Minglu Zhao

4 May 2023 in International Journal of Multimedia Information Retrieval
Verified
Journal Article
Closed access
PDF file
RLRuochen Li
Ruochen Li
NLNannan Li
Nannan Li
WWWenmin Wang
Wenmin Wang

27 October 2026 in Neurocomputing
Verified
Journal Article
Closed access
HHHuanqiang Hu
Huanqiang Hu
NLNannan Li
Nannan Li
RLRuochen Li
Ruochen Li
QWQing Wang
Qing Wang
LFLei Fu
Lei Fu
ZCZhaowen Chen
Zhaowen Chen