An Yan

An Yan

Auto-generated profile, based on publicly available information

An Yan

UC San Diego La Jolla 92037 CA

Research Focus & Metrics

Academic Career Length

7Years (2020-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 January 2023
Verified
Conference Paper
Open access
PDF file
WZWanrong Zhu
Wanrong Zhu
AYAn Yan
An Yan
YLYujie Lu
Yujie Lu
WXWenda Xu
Wenda Xu
XWXin Wang
Xin Wang
WWWilliam Yang Wang
William Yang Wang
Wanrong Zhu, An Yan, Yujie Lu, Wenda Xu, Xin Wang, Miguel Eckstein, William Yang Wang. Findings of the Association for Computational Linguistics: EACL 2023. 2023.

6 December 2020 in arXiv
Verified
Preprint
Open access
PDF file
AYAn Yan
An Yan
XWXin Eric Wang
Xin Eric Wang
JFJiangtao Feng
Jiangtao Feng
LLLei Li
Lei Li
WWWilliam Yang Wang
William Yang Wang
Commanding a robot to navigate with natural language instructions is a long-term goal for grounded language understanding and robotics. But the dominant language is English, according to previous studies on vision-language navigation (VLN). To go beyond English and serve people speaking different languages, we collect a bilingual Room-to-Room (BL-R2R) dataset, extending the original benchmark with new Chinese instructions. Based on this newly introduced dataset, we study how an agent can be trained on existing English instructions but navigate effectively with another language under a zero-shot learning scenario. Without any training data of the target language, our model shows competitive results even compared to a model with full access to the target language training data. Moreover, we investigate the transferring ability of our model when given a certain amount of target language training data.

13 November 2023 in arXiv
Verified
Preprint
Open access
PDF file
AYAn Yan
An Yan
ZYZhengyuan Yang
Zhengyuan Yang
WZWanrong Zhu
Wanrong Zhu
KLKevin Lin
Kevin Lin
LLLinjie Li
Linjie Li
LWLijuan Wang
Lijuan Wang
We present MM-Navigator, a GPT-4V-based agent for the smartphone graphical user interface (GUI) navigation task. MM-Navigator can interact with a smartphone screen as human users, and determine subsequent actions to fulfill given instructions. Our findings demonstrate that large multimodal models (LMMs), specifically GPT-4V, excel in zero-shot GUI navigation through its advanced screen interpretation, action reasoning, and precise action localization capabilities. We first benchmark MM-Navigator on our collected iOS screen dataset. According to human assessments, the system exhibited a 91\% accuracy rate in generating reasonable action descriptions and a 75\% accuracy rate in executing the correct actions for single-step instructions on iOS. Additionally, we evaluate the model on a subset of an Android screen navigation dataset, where the model outperforms previous GUI navigators in a zero-shot fashion. Our benchmark and detailed analyses aim to lay a robust groundwork for future research into the GUI navigation task. The project page is at https://github.com/zzxslp/MM-Navigator.

1 January 2023
Verified
Conference Paper
Open access
PDF file
WZWanrong Zhu
Wanrong Zhu
XWXin Wang
Xin Wang
AYAn Yan
An Yan
MEMiguel Eckstein
Miguel Eckstein
WWWilliam Yang Wang
William Yang Wang
Wanrong Zhu, Xin Wang, An Yan, Miguel Eckstein, William Yang Wang. Findings of the Association for Computational Linguistics: EACL 2023. 2023.

15 February 2023 in arXiv
Verified
Preprint
Open access
PDF file
WZWanrong Zhu
Wanrong Zhu
XWXin Eric Wang
Xin Eric Wang
AYAn Yan
An Yan
MEMiguel Eckstein
Miguel Eckstein
WWWilliam Yang Wang
William Yang Wang
Automatic evaluations for natural language generation (NLG) conventionally rely on token-level or embedding-level comparisons with text references. This differs from human language processing, for which visual imagination often improves comprehension. In this work, we propose ImaginE, an imagination-based automatic evaluation metric for natural language generation. With the help of StableDiffusion, a state-of-the-art text-to-image generator, we automatically generate an image as the embodied imagination for the text snippet and compute the imagination similarity using contextual embeddings. Experiments spanning several text generation tasks demonstrate that adding machine-generated images with our ImaginE displays great potential in introducing multi-modal information into NLG evaluation, and improves existing automatic metrics' correlations with human similarity judgments in both reference-based and reference-free evaluation scenarios.

23 July 2023 in arXiv
Verified
Preprint
Open access
PDF file
JEJessica Echterhoff
Jessica Echterhoff
AYAn Yan
An Yan
JMJulian J. McAuley
Julian J. McAuley
It is time-consuming to find the best product among many similar alternatives. Comparative sentences can help to contrast one item from others in a way that highlights important features of an item that stand out. Given reviews of one or multiple items and relevant item features, we generate comparative review sentences to aid users to find the best fit. Specifically, our model consists of three successive components in a transformer: (i) an item encoding module to encode an item for comparison, (ii) a comparison generation module that generates comparative sentences in an autoregressive manner, (iii) a novel decoding method for user personalization. We show that our pipeline generates fluent and diverse comparative sentences. We run experiments on the relevance and fidelity of our generated sentences in a human evaluation study and find that our algorithm creates comparative review sentences that are relevant and truthful.