Zhibo Yang

Zhibo Yang

Auto-generated profile, based on publicly available information

Zhibo Yang

Alibaba Group, Hangzhou, China

Research Focus & Metrics

Academic Career Length

6Years (2021-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
24 March 2026 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Verified
Journal Article
Closed access
WYWenwen Yu
Wenwen Yu
ZYZhibo Yang
Zhibo Yang
JWJianqiang Wan
Jianqiang Wan
SSSibo Song
Sibo Song
JTJun Tang
Jun Tang
XBXiang Bai
Xiang Bai
Visually-situated text parsing (VsTP) has recently seen notable advancements, driven by the growing demand for automated document understanding and the emergence of large language models capable of processing document-based questions. While various methods have been proposed to tackle the complexities of VsTP, existing solutions often rely on task-specific architectures and objectives for individual tasks. This leads to modal isolation and complex workflows due to the diversified targets and heterogeneous schemas. In this paper, we introduce OmniParser V2, a universal model that unifies VsTP typical tasks, including text spotting, key information extraction, table recognition, and layout analysis, into a unified framework. Central to our approach is the proposed Structured-Points-of-Thought (SPOT) prompting schemas, which improves model performance across diverse scenarios by leveraging a unified encoder-decoder architecture, objective, and input&output representation. SPOT eliminates the need for task-specific architectures and loss functions, significantly simplifying the processing pipeline. Our extensive evaluations across four tasks on eight different datasets show that OmniParser V2 achieves state-of-the-art or competitive results in VsTP. Additionally, we explore the integration of SPOT within a multimodal large language model structure, further enhancing visual text parsing capabilities on four tasks, thereby confirming the generality of SPOT prompting technique. The code is available at https://github.com/AlibabaResearch/AdvancedLiterateMachinery.

31 August 2024 in Pattern Recognition
Verified
Journal Article
Closed access
YZYuyi Zhang
Yuyi Zhang
YZYuanzhi Zhu
Yuanzhi Zhu
DPDezhi Peng
Dezhi Peng
PZPeirong Zhang
Peirong Zhang
ZYZhenhua Yang
Zhenhua Yang
LJLianwen Jin
Lianwen Jin

23 July 2024 in Pattern Recognition
Verified
Journal Article
Closed access
RLRujiao Long
Rujiao Long
HXHangdi Xing
Hangdi Xing
ZYZhibo Yang
Zhibo Yang
QZQi Zheng
Qi Zheng
ZYZhi Yu
Zhi Yu
CYCong Yao
Cong Yao

1 December 2024
Verified
Book Chapter
Closed access
PDF file
YZYuanzhi Zhu
Yuanzhi Zhu
JLJiawei Liu
Jiawei Liu
FGFeiyu Gao
Feiyu Gao
WLWenyu Liu
Wenyu Liu
Xinggang WangXinggang Wang
Xinggang Wang
ZYZhibo Yang
Zhibo Yang

20 January 2026 in IEEE Transactions on Circuits and Systems for Video Technology
Verified
Journal Article
Closed access
JLJiawei Liu
Jiawei Liu
YZYuanzhi Zhu
Yuanzhi Zhu
FGFeiyu Gao
Feiyu Gao
ZYZhibo Yang
Zhibo Yang
PWPeng Wang
Peng Wang
WLWenyu Liu
Wenyu Liu
Generating visual text in natural scene images is a challenging task with many unsolved problems. Different from generating text on artificially designed images (such as posters, covers, and cartoons), existing methods for natural scene visual text generation still have significant deficiencies: methods based on rendering engines rely on manually crafted rules, which struggle to adapt to diverse backgrounds and leave obvious artificial traces, while their text layouts may be placed in unreasonable areas (e.g., sky or ground) and text content is semantically disconnected from the scene; diffusion model-based methods, on the other hand, face difficulties in generating small characters, depend on manually designed prompts to ensure reasonable layout and content, fail to generate text at precise locations, and cannot effectively control text attributes (e.g., font and color). In this paper, we propose a two-stage method named SceneVTG++ to address these issues. SceneVTG++ comprises two core components: a Text Layout and Content Generator (TLCG) and a Controllable Local Text Diffusion (CLTD). The former leverages the world knowledge and visual reasoning capabilities of multimodal large language models to identify reasonable text areas and recommend scene-relevant text content based on natural scene background images; the latter generates controllable multilingual text using a diffusion model, ensuring alignment with the outputs of TLCG. Through extensive experiments, we verified the effectiveness of both TLCG and CLTD, and demonstrated that SceneVTG++ achieves state-of-the-art performance in natural scene visual text generation. Additionally, the images generated by SceneVTG++ exhibit superior utility for training natural scene optical character recognition (OCR) tasks, including text detection and text recognition. Codes and datasets will be made publicly available.

28 October 2024
Verified
Conference Paper
Open access
HZHumen Zhong
Humen Zhong
ZYZhiBo Yang
ZhiBo Yang
ZLZhaohai Li
Zhaohai Li
PWPeng Wang
Peng Wang
JTJun Tang
Jun Tang
CYCong Yao
Cong Yao
Text recognition is an inherent integration of vision and language, encompassing the visual texture in stroke patterns and the semantic context among the character sequences. Towards advanced text recognition, there are three key challenges: (1) an encoder capable of representing the visual and semantic distributions; (2) a decoder that ensures the alignment between vision and semantics; and (3) consistency in the framework during pre-training, if it exists, and fine-tuning. Inspired by masked autoencoding, a successful pre-training strategy in both vision and language, we propose an innovative scene text recognition approach, named VL-Reader. The novelty of the VL-Reader lies in the pervasive interplay between vision and language throughout the entire process. Concretely, we first introduce a Masked Visual-Linguistic Reconstruction (MVLR) objective, which aims at simultaneously modeling visual and linguistic information. Then, we design a Masked Visual-Linguistic Decoder (MVLD) to further leverage masked vision-language context and achieve bi-modal feature interaction. The architecture of VL-Reader maintains consistency from pre-training to fine-tuning. In the pre-training stage, VL-Reader reconstructs both masked visual and text tokens, while in the fine-tuning stage, the network degrades to reconstruct all characters from an image without any masked regions. VL-reader achieves an average accuracy of 97.1% on six typical datasets, surpassing the SOTA by 1.1%. The improvement was even more significant on challenging datasets. The results demonstrate that vision and language reconstructor can serve as an effective scene text recognizer.