Jiwen Zhang

Ms. Jiwen Zhang

orcid

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 January 2025
Verified
Conference Paper
Open access
PDF file
JZJiwen Zhang
Jiwen Zhang
YYYa-Qi Yu
Ya-Qi Yu
MLMinghui Liao
Minghui Liao
WLWenTao Li
WenTao Li
JWJihao Wu
Jihao Wu
ZWZhongyu Wei
Zhongyu Wei
Jiwen Zhang, Ya-Qi Yu, Minghui Liao, WenTao Li, Jihao Wu, Zhongyu Wei. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

28 October 2024
Verified
Conference Paper
Open access
ZLZejun Li
Zejun Li
YWYe Wang
Ye Wang
MDMengfei Du
Mengfei Du
QLQingwen Liu
Qingwen Liu
BWBinhao Wu
Binhao Wu
XHXuanjing Huang
Xuanjing Huang
Recent years have witnessed remarkable progress in the development of large vision-language models (LVLMs). Benefiting from the strong language backbones and efficient cross-modal alignment strategies, LVLMs exhibit surprising capabilities to perceive visual signals and perform visually grounded reasoning. However, the capabilities of LVLMs have not been comprehensively and quantitatively evaluated. Most existing multi-modal benchmarks require task-oriented input-output formats, posing great challenges to automatically assess the free-form text output of LVLMs. To effectively leverage the annotations available and reduce the manual efforts required for constructing new benchmarks, we propose to re-formulate existing benchmarks into unified LVLM-compatible formats. Through systematic data collection and reformulation, we present ReForm-Eval benchmark, offering substantial data for evaluating various capabilities of LVLMs. Through extensive experiments and analysis in ReForm-Eval, we demonstrate the comprehensiveness and reliability of ReForm-Eval in assessing various LVLMs. Our benchmark and evaluation framework is now available at https://github.com/FudanDISC/ReForm-Eval

11 November 2024 in Preprints.org
Verified
Preprint
Open access
PDF file
ZLZejun Li
Zejun Li
JZJiwen Zhang
Jiwen Zhang
DWDianyi Wang
Dianyi Wang
YWYe Wang
Ye Wang
XHXuanjing Huang
Xuanjing Huang
ZWZhongyu Wei
Zhongyu Wei
With the success of large language models (LLMs) driving progress towards general-purpose AI, there has been a growing focus on extending these models to multi-modal domains, giving rise to large multi-modal models (LMMs). Unlike existing reviews that focuses on specific model frameworks or scenarios, this survey summarizes and provides insights into the current research on LMMs from a more general perspective, \textbf{input-output space extension}. Particularly, we discuss the following questions: (i) How to construct multi-modal input-output spaces with discretely or continuously encoded modality signals? (ii) How to design model architectures and corresponding training strategies to align the constructed multi-modal representation space? (iii) How to comprehensively evaluate LMMs based on the expanded input-output space? We hope to provide an intuitive and comprehensive overview and inspire future work.

27 May 2025 in arXiv
Verified
Preprint
Open access
PDF file
XDXuanwen Ding
Xuanwen Ding
CPChengjun Pan
Chengjun Pan
ZLZejun Li
Zejun Li
JZJiwen Zhang
Jiwen Zhang
SWSiyuan Wang
Siyuan Wang
ZWZhongyu Wei
Zhongyu Wei
Evaluating multimodal large language models (MLLMs) is increasingly expensive, as the growing size and cross-modality complexity of benchmarks demand significant scoring efforts. To tackle with this difficulty, we introduce AutoJudger, an agent-driven framework for efficient and adaptive benchmarking of MLLMs that tackles this escalating cost. AutoJudger employs the Item Response Theory (IRT) to estimate the question difficulty and an autonomous evaluation agent to dynamically select the most informative test questions based on the model's real-time performance. Specifically, AutoJudger incorporates two pivotal components: a semantic-aware retrieval mechanism to ensure that selected questions cover diverse and challenging scenarios across both vision and language modalities, and a dynamic memory that maintains contextual statistics of previously evaluated questions to guide coherent and globally informed question selection throughout the evaluation process. Extensive experiments on four representative multimodal benchmarks demonstrate that our adaptive framework dramatically reduces evaluation expenses, i.e. AutoJudger uses only 4% of the data to achieve over 90% ranking accuracy with the full benchmark evaluation on MMT-Bench.

16 July 2023 in arXiv
Verified
Preprint
Open access
PDF file
RLRuipu Luo
Ruipu Luo
JZJiwen Zhang
Jiwen Zhang
ZWZhongyu Wei
Zhongyu Wei
Vision language decision making (VLDM) is a challenging multimodal task. The agent have to understand complex human instructions and complete compositional tasks involving environment navigation and object manipulation. However, the long action sequences involved in VLDM make the task difficult to learn. From an environment perspective, we find that task episodes can be divided into fine-grained \textit{units}, each containing a navigation phase and an interaction phase. Since the environment within a unit stays unchanged, we propose a novel hybrid-training framework that enables active exploration in the environment and reduces the exposure bias. Such framework leverages the unit-grained configurations and is model-agnostic. Specifically, we design a Unit-Transformer (UT) with an intrinsic recurrent state that maintains a unit-scale cross-modal memory. Through extensive experiments on the TEACH benchmark, we demonstrate that our proposed framework outperforms existing state-of-the-art methods in terms of all evaluation metrics. Overall, our work introduces a novel approach to tackling the VLDM task by breaking it down into smaller, manageable units and utilizing a hybrid-training framework. By doing so, we provide a more flexible and effective solution for multimodal decision making.

30 August 2024 in Preprints.org
Verified
Preprint
Open access
PDF file
JZJiwen Zhang
Jiwen Zhang
YYYaqi Yu
Yaqi Yu
MLMinghui Liao
Minghui Liao
WLWentao Li
Wentao Li
JWJihao Wu
Jihao Wu
ZWZhongyu Wei
Zhongyu Wei
Graphical User Interface (GUI) agents are expected to precisely operate on the screens of digital devices. Existing GUI agents merely rely on current visual observations and plain-text action history, ignoring the significance of history screens. To mitigate this issue, we propose UI-Hawk, a visual GUI agent specially designed to processing screen streams encountered during GUI navigation. UI-Hawk incorporates a history-aware visual encoder and an efficient resampler to handle the screen sequences. To acquire a better understanding of screen streams, we define four fundamental tasks—UI grounding, UI referring, screen question answering, and screen summarization. We develop an automated data curation method to generate the corresponding training data for UI-Hawk. Along with the efforts above, we have also created a benchmark FunUI to quantitatively evaluate the fundamental screen understanding ability of MLLMs. Extensive experiments on FunUI and GUI navigation benchmarks consistently validate that screen stream understanding is not only beneficial but also essential for GUI navigation.