Haichao Zhang

Haichao Zhang


Research Focus & Metrics

Representative Publications

View All Publications
26 October 2023
Verified
Conference Paper
Open access
HZHaichao Zhang
Haichao Zhang
YXYi Xu
Yi Xu
HLHongsheng Lu
Hongsheng Lu
STShimizu Takayuki
Shimizu Takayuki
YFYun Fu
Yun Fu
Trajectory prediction plays a vital role in understanding pedestrian movement for applications such as autonomous driving and robotics. Current trajectory prediction models depend on long, complete, and accurately observed sequences from visual modalities. Nevertheless, real-world situations often involve obstructed cameras, missed objects, or objects out of sight due to environmental factors, leading to incomplete or noisy trajectories. To overcome these limitations, we propose LTrajDiff, a novel approach that treats objects obstructed or out of sight as equally important as those with fully visible trajectories. LTrajDiff utilizes sensor data from mobile phones to surmount out-of-sight constraints, albeit introducing new challenges such as modality fusion, noisy data, and the absence of spatial layout and object size information. We employ a denoising diffusion model to predict precise layout sequences from noisy mobile data using a coarse-to-fine diffusion strategy, incorporating the Random Mask Strategy, Siamese Masked Encoding Module, and Modality Fusion Module. Our model predicts layout sequences by implicitly inferring object size and projection status from a single reference timestamp or significantly obstructed sequences. Achieving state-of-the-art results in randomly obstructed experiments, our model outperforms other baselines in extremely short input experiments, illustrating the effectiveness of leveraging noisy mobile data for layout sequence prediction. In summary, our approach offers a promising solution to the challenges faced by layout sequence and trajectory prediction models in real-world settings, paving the way for utilizing sensor data from mobile phones to accurately predict pedestrian bounding box trajectories. To the best of our knowledge, this is the first work that addresses severely obstructed and extremely short layout sequences by combining vision with noisy mobile modality, making it the pioneering work in the field of layout sequence trajectory prediction.

23 March 2026 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Verified
Journal Article
Closed access
HZHaichao Zhang
Haichao Zhang
YXYi Xu
Yi Xu
YFYun Fu
Yun Fu
Trajectory prediction is a fundamental problem in computer vision, vision-language-action models, world models, and autonomous systems, with broad impact on applications including autonomous driving, robotics, and surveillance. Most existing approaches assume observations are complete and relatively clean, and thus do not adequately address out-ofsight agents or the intrinsic noise in sensing modalities (e.g., sensor measurements) caused by restricted camera coverage, occlusions, and the lack of ground-truth denoised trajectories. These factors introduce substantial safety concerns and reduce the robustness of trajectory prediction in practical deployments. In this extended study, we introduce major improvements to Out-of-Sight Trajectory (OST), a new task aimed at predicting noise-free visual trajectories of out-of-sight objects from noisy sensor observations. Based on our prior work, we expand the setting of Out-of-Sight Trajectory Prediction (OOSTraj) from pedestrians to both pedestrians and vehicles, thereby increasing its relevance to autonomous driving, robotics, and surveillance scenarios. Our improved Vision-Positioning Denoising Module utilizes camera calibration to construct a vision-position correspondence, mitigating the absence of direct visual cues while enabling effective unsupervised denoising of noisy sensor signals. Extensive experiments on the Vi-Fi and JRDB datasets demonstrate that our method achieves state-of-the-art results for both trajectory denoising and trajectory prediction, with clear gains over prior baselines. We further provide comparisons against classical denoising techniques, including Kalman filtering, and adapt recent trajectory prediction models to this setting, establishing a stronger and more comprehensive benchmark. To the best of our knowledge, this is the first work to incorporate vision-positioning projection to denoise noisy sensor trajectories of out-of-sight agents, opening new directions for future research in this area. The code and preprocessed datasets are available at https://github.com/Hai-chao-Zhang/OST.

2 June 2026 in Preprints.org
Verified
Preprint
Open access
PDF file
HZHaichao Zhang
Haichao Zhang
MCMingfei Chen
Mingfei Chen
SHShwai He
Shwai He
ZXZhengtong Xu
Zhengtong Xu
YSYifan Shen
Yifan Shen
YFYun Fu
Yun Fu
Physical AI aims to extend artificial intelligence from digital reasoning to perception, prediction, simulation, planning, and action in the physical world. While recent progress has advanced through vision-language models, vision-language-action models, world models, policy learning, and embodied agents, existing discussions are often organized from robotics-centric, vision-centric, or cyber-physical perspectives. This survey instead studies Physical AI through the lens of LLM-based world knowledge. We argue that LLMs encode implicit semantic, commonsense, procedural, and causal priors through large-scale pretraining, making them useful high-level sources for physical reasoning, multimodal grounding, action grounding, and embodied decision making. However, language-mediated knowledge is sparse and lossy for dense physical states, such as geometry, motion, contact, force, high-frequency dynamics, and long-horizon temporal evolution. VLMs and MLLMs ground LLM-derived priors into perception, but often expose physical understanding through language outputs. VLAs connect perception and language to executable actions, yet typically lack predictive models of how the world evolves under actions. This motivates a roadmap from LLM-based world knowledge to multimodal grounding, action grounding, world modeling, policy learning, and embodied deployment. We review how these components provide perceptual, actionable, predictive, and simulative substrates for Physical AI, and discuss open challenges in grounding, world modeling, closed-loop evaluation, safety, and generalization. A curated repository of related papers and resources is available at https://github.com/Hai-chao-Zhang/Awesome-Physical-AI.

18 September 2025 in arXiv
Verified
Preprint
Open access
PDF file
HZHaichao Zhang
Haichao Zhang
WCWenhao Chai
Wenhao Chai
SHShwai He
Shwai He
ALANG Li
ANG Li
YFYun Fu
Yun Fu
High temporal resolution is essential for capturing fine-grained details in video understanding. However, current video large language models (VLLMs) and benchmarks mostly rely on low-frame-rate sampling, such as uniform sampling or keyframe selection, discarding dense temporal information. This compromise avoids the high cost of tokenizing every frame, which otherwise leads to redundant computation and linear token growth as video length increases. While this trade-off works for slowly changing content, it fails for tasks like lecture comprehension, where information appears in nearly every frame and requires precise temporal alignment. To address this gap, we introduce Dense Video Understanding (DVU), which enables high-FPS video comprehension by reducing both tokenization time and token overhead. Existing benchmarks are also limited, as their QA pairs focus on coarse content changes. We therefore propose DIVE (Dense Information Video Evaluation), the first benchmark designed for dense temporal reasoning. To make DVU practical, we present Gated Residual Tokenization (GRT), a two-stage framework: (1) Motion-Compensated Inter-Gated Tokenization uses pixel-level motion estimation to skip static regions during tokenization, achieving sub-linear growth in token count and compute. (2) Semantic-Scene Intra-Tokenization Merging fuses tokens across static regions within a scene, further reducing redundancy while preserving dynamic semantics. Experiments on DIVE show that GRT outperforms larger VLLM baselines and scales positively with FPS. These results highlight the importance of dense temporal information and demonstrate that GRT enables efficient, scalable high-FPS video understanding.

1 October 2026
Verified
Book Chapter
Closed access
PDF file
HZHaichao Zhang
Haichao Zhang
WCWenhao Chai
Wenhao Chai
SHShwai He
Shwai He
ALANG Li
ANG Li
YFYun Fu
Yun Fu

1 January 2022 in SSRN Electronic Journal
Verified
Preprint
Open access
PDF file
WXWei Xu
Wei Xu
YSYecun Shi
Yecun Shi
YXYu Xi
Yu Xi
JLJingyue Liu
Jingyue Liu
TWTan Wang
Tan Wang
JYJianquan Yao
Jianquan Yao
Due to the particular screen printing rules defined by the unique chip manufacturer and the huge variety of chips, it is difficult to identify chip information by naked eye in chip reliability detection. Automatic identification character technology can help us obtain the information of the chip. But the special character samples in the dataset is extremely uneven. When recognizing characters on chip surface by deep learning, the limited certain characters dataset will reduce the recognition accuracy. A character recognition method for chip surface characters based on text replacement is proposed in this paper. Firstly, the efficient channel attention modules are added to enhance the feature extraction of characters in the text style transfer process. Secondly, the dilated convolution layers are used to replace the residual module during the background restoration process, which help to obtain a more comprehensive background texture feature. Finally, a one-stage target detection network YOLOv5s is used to detect the characters on the chip surface. The experimental results demonstrated that the three evaluation metrics for image generation quality (l2 error, psnr, and ssim) achieved better performance than previous work due to the text replacement network. The overall charac-