Haoyu Cao

Haoyu Cao

Auto-generated profile, based on publicly available information

Haoyu Cao

University of Science and Technology of China

Research Focus & Metrics

Academic Career Length

3Years (2024-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
20 March 2024 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Verified
Journal Article
Closed access
WYWenwen Yu
Wenwen Yu
YLYuliang Liu
Yuliang Liu
XZXingkui Zhu
Xingkui Zhu
HCHaoyu Cao
Haoyu Cao
XSXing Sun
Xing Sun
XBXiang Bai
Xiang Bai
We exploit the potential of the large-scale Contrastive Language-Image Pretraining (CLIP) model to enhance scene text detection and spotting tasks, transforming it into a robust backbone, FastTCM-CR50. This backbone utilizes visual prompt learning and cross-attention in CLIP to extract image and text-based prior knowledge. Using predefined and learnable prompts, FastTCM-CR50 introduces an instance-language matching process to enhance the synergy between image and text embeddings, thereby refining text regions. Our Bimodal Similarity Matching (BSM) module facilitates dynamic language prompt generation, enabling offline computations and improving performance. FastTCM-CR50 offers several advantages: 1) It can enhance existing text detectors and spotters, improving performance by an average of 1.6% and 1.5%, respectively. 2) It outperforms the previous TCM-CR50 backbone, yielding an average improvement of 0.2% and 0.55% in text detection and spotting tasks, along with a 47.1% increase in inference speed. 3) It showcases robust few-shot training capabilities. Utilizing only 10% of the supervised data, FastTCM-CR50 improves performance by an average of 26.5% and 4.7% for text detection and spotting tasks, respectively. 4) It consistently enhances performance on out-of-distribution text detection and spotting datasets, particularly the NightTime-ArT subset from ICDAR2019-ArT and the DOTA dataset for oriented object detection. The code is available at https://github.com/wenwenyu/TCM .

28 October 2024
Verified
Conference Paper
Open access
YWYubo Wang
Yubo Wang
CLChaohu Liu
Chaohu Liu
YQYanqiu Qu
Yanqiu Qu
HCHaoyu Cao
Haoyu Cao
DJDeqiang Jiang
Deqiang Jiang
LXLinli Xu
Linli Xu
Large vision-language models (LVLMs) integrate visual information into large language models, showcasing remarkable multi-modal conversational capabilities. However, the visual modules introduces new challenges in terms of robustness for LVLMs, as attackers can craft adversarial images that are visually clean but may mislead the model to generate incorrect answers. In general, LVLMs rely on vision encoders to transform images into visual tokens, which are crucial for the language models to perceive image contents effectively. Therefore, we are curious about one question: Can LVLMs still generate correct responses when the encoded visual tokens are attacked and disrupting the visual information? To this end, we propose a non-targeted attack method referred to as VT-Attack (Visual Tokens Attack), which constructs adversarial examples from multiple perspectives, with the goal of comprehensively disrupting feature representations and inherent relationships as well as the semantic properties of visual tokens output by image encoders. Using only access to the image encoder in the proposed attack, the generated adversarial examples exhibit transferability across diverse LVLMs utilizing the same image encoder and generality across different tasks. Extensive experiments validate the superior attack performance of the VT-Attack over baseline methods, demonstrating its effectiveness in attacking LVLMs with image encoders, which in turn can provide guidance on the robustness of LVLMs, particularly in terms of the stability of the visual feature space.

24 October 2025 in IEEE Transactions on Circuits and Systems for Video Technology
Verified
Journal Article
Closed access
CLChaohu Liu
Chaohu Liu
YWYubo Wang
Yubo Wang
HCHaoyu Cao
Haoyu Cao
BLBing Liu
Bing Liu
DJDeqiang Jiang
Deqiang Jiang
Vision-language models (VLMs), such as BLIP-2 and LLaVA, have significantly advanced multimodal understanding but exhibit critical vulnerabilities to visual adversarial perturbations. The high efficacy of untargeted attacks, in particular, poses significant concerns for their operational robustness. Conventional attack methods generate adversarial examples by backpropagating the language modeling loss from the final output to the input image. However, the deep architecture of the integrated large language models (LLMs) often diminishes this gradient flow, limiting the attack’s effectiveness in perturbing the visual domain. To address this limitation, we introduce a novel untargeted attack method based on Maximizing Information Entropy (MIE). Our approach enhances attack efficacy not only by maximizing the information entropy of the model’s final output but also by directly inducing uncertainty within its internal feature representations. This layer-wise perturbation strategy disrupts the model’s cognitive process more comprehensively than relying on the final output layer alone. We provide a theoretical analysis demonstrating that the uncertainty induced by MIE is greater than or equal to that of conventional output-only attacks. Comprehensive quantitative evaluations across multiple VLM architectures and datasets confirm that our method consistently outperforms existing techniques, thereby establishing a new, more rigorous benchmark for assessing adversarial robustness in vision-language models.

27 October 2025
Verified
Conference Paper
Open access
XLXin Li
Xin Li
MGMingming Gong
Mingming Gong
YWYunfei Wu
Yunfei Wu
JDJianxin Dai
Jianxin Dai
AGAntai Guo
Antai Guo
XSXing Sun
Xing Sun

27 October 2025
Verified
Conference Paper
Open access
YHYongXiang Hua
YongXiang Hua
HCHaoyu Cao
Haoyu Cao
ZTZhou Tao
Zhou Tao
BLBocheng Li
Bocheng Li
ZWZihao Wu
Zihao Wu
LXLinli Xu
Linli Xu

9 November 2026
Verified
Conference Paper
Open access
CLChaohu Liu
Chaohu Liu
DSDesong Sui
Desong Sui
HCHaoyu Cao
Haoyu Cao
YHYongXiang Hua
YongXiang Hua
LXLinli Xu
Linli Xu