Zhiyang Xu

Zhiyang Xu

Auto-generated profile, based on publicly available information

Zhiyang Xu

Virginia Tech

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 January 2023
Verified
Conference Paper
Open access
PDF file
JQJingyuan Qi
Jingyuan Qi
ZXZhiyang Xu
Zhiyang Xu
YSYing Shen
Ying Shen
MLMinqian Liu
Minqian Liu
DJDi Jin
Di Jin
LHLifu Huang
Lifu Huang
Jingyuan Qi, Zhiyang Xu, Ying Shen, Minqian Liu, Di Jin, Qifan Wang, Lifu Huang. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

1 January 2024
Verified
Conference Paper
Open access
PDF file
YSYing Shen
Ying Shen
ZXZhiyang Xu
Zhiyang Xu
QWQifan Wang
Qifan Wang
YCYu Cheng
Yu Cheng
WYWenpeng Yin
Wenpeng Yin
LHLifu Huang
Lifu Huang
Ying Shen, Zhiyang Xu, Qifan Wang, Yu Cheng, Wenpeng Yin, Lifu Huang. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

1 January 2024
Verified
Conference Paper
Open access
PDF file
MLMinqian Liu
Minqian Liu
YSYing Shen
Ying Shen
ZXZhiyang Xu
Zhiyang Xu
YCYixin Cao
Yixin Cao
ECEunah Cho
Eunah Cho
LHLifu Huang
Lifu Huang
Minqian Liu, Ying Shen, Zhiyang Xu, Yixin Cao, Eunah Cho, Vaibhav Kumar, Reza Ghanadan, Lifu Huang. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

1 January 2025
Verified
Conference Paper
Open access
PDF file
DMDehai Min
Dehai Min
ZXZhiyang Xu
Zhiyang Xu
Guilin QiGuilin Qi
Guilin Qi
LHLifu Huang
Lifu Huang
CYChenyu You
Chenyu You
Dehai Min, Zhiyang Xu, Guilin Qi, Lifu Huang, Chenyu You. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

24 March 2024 in Proceedings of the AAAI Conference on Artificial Intelligence
Verified
Journal Article
Open access
PDF file
JQJingyuan Qi
Jingyuan Qi
MLMinqian Liu
Minqian Liu
YSYing Shen
Ying Shen
ZXZhiyang Xu
Zhiyang Xu
LHLifu Huang
Lifu Huang
Automatically generating scripts (i.e. sequences of key steps described in text) from video demonstrations and reasoning about the subsequent steps are crucial to the modern AI virtual assistants to guide humans to complete everyday tasks, especially unfamiliar ones. However, current methods for generative script learning rely heavily on well-structured preceding steps described in text and/or images or are limited to a certain domain, resulting in a disparity with real-world user scenarios. To address these limitations, we present a new benchmark challenge – MULTISCRIPT, with two new tasks on task-oriented multimodal script learning: (1) multimodal script generation, and (2) subsequent step prediction. For both tasks, the input consists of a target task name and a video illustrating what has been done to complete the target task, and the expected output is (1) a sequence of structured step descriptions in text based on the demonstration video, and (2) a single text description for the subsequent step, respectively. Built from WikiHow, MULTISCRIPT covers multimodal scripts in videos and text descriptions for over 6,655 human everyday tasks across 19 diverse domains. To establish baseline performance on MULTISCRIPT, we propose two knowledge-guided multimodal generative frameworks that incorporate the task-related knowledge prompted from large language models such as Vicuna. Experimental results show that our proposed approaches significantly improve over the competitive baselines.

10 June 2023 in arXiv
Verified
Preprint
Open access
PDF file
ZXZhiyang Xu
Zhiyang Xu
YSYing Shen
Ying Shen
LHLifu Huang
Lifu Huang
Instruction tuning, a new learning paradigm that fine-tunes pre-trained language models on tasks specified through instructions, has shown promising zero-shot performance on various natural language processing tasks. However, it has yet to be explored for vision and multimodal tasks. In this work, we introduce MUL-TIINSTRUCT, the first multimodal instruction tuning benchmark dataset that consists of 62 diverse multimodal tasks in a unified seq-to-seq format covering 10 broad categories. The tasks are derived from 21 existing open-source datasets and each task is equipped with 5 expert-written instructions. We take OFA as the base pre-trained model for multimodal instruction tuning, and to further improve its zero-shot performance, we explore multiple transfer learning strategies to leverage the large-scale NATURAL INSTRUCTIONS dataset. Experimental results demonstrate strong zero-shot performance on various unseen multimodal tasks and the benefit of transfer learning from a text-only instruction dataset. We also design a new evaluation metric - Sensitivity, to evaluate how sensitive the model is to the variety of instructions. Our results indicate that fine-tuning the model on a diverse set of tasks and instructions leads to a reduced sensitivity to variations in instructions for each task.