Hengshuang Zhao

Hengshuang Zhao

Auto-generated profile, based on publicly available information

Hengshuang Zhao

The University of Hong Kong

Research Focus & Metrics

Academic Career Length

10Years (2017-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
7 October 2018
Verified
Book Chapter
Closed access
PDF file
HZHengshuang Zhao
Hengshuang Zhao
XQXiaojuan Qi
Xiaojuan Qi
XSXiaoyong Shen
Xiaoyong Shen
JSJianping Shi
Jianping Shi
JJJiaya Jia
Jiaya Jia

5 October 2018
Verified
Book Chapter
Closed access
PDF file
HZHengshuang Zhao
Hengshuang Zhao
YZYi Zhang
Yi Zhang
SLShu Liu
Shu Liu
JSJianping Shi
Jianping Shi
CLChen Change Loy
Chen Change Loy
JJJiaya Jia
Jiaya Jia

7 August 2024 in IEEE Robotics and Automation Letters
Verified
Journal Article
Closed access
ZXZhenhua Xu
Zhenhua Xu
YZYujia Zhang
Yujia Zhang
EXEnze Xie
Enze Xie
ZZZhen Zhao
Zhen Zhao
YGYong Guo
Yong Guo
HZHengshuang Zhao
Hengshuang Zhao
Multimodallarge language models (MLLMs) have emerged as a prominent area of interest within the research community, given their proficiency in handling and reasoning with non-textual data, including images and videos. This study seeks to extend the application of MLLMs to the realm of autonomous driving by introducing DriveGPT4, a novel interpretable end-to-end autonomous driving system based on LLMs. Capable of processing multi-frame video inputs and textual queries, DriveGPT4 facilitates the interpretation of vehicle actions, offers pertinent reasoning, and effectively addresses a diverse range of questions posed by users. Furthermore, DriveGPT4 predicts low-level vehicle control signals in an end-to-end fashion. These advanced capabilities are achieved through the utilization of a bespoke visual instruction tuning dataset, specifically tailored for autonomous driving applications, in conjunction with a mix-finetuning training strategy. DriveGPT4 represents the pioneering effort to leverage LLMs for the development of an interpretable end-to-end autonomous driving solution. Evaluations conducted on the BDD-X dataset showcase the superior qualitative and quantitative performance of DriveGPT4. Additionally, the fine-tuning of domain-specific data enables DriveGPT4 to yield close or even improved results in terms of autonomous driving grounding when contrasted with GPT4-V.

15 February 2023 in International Journal of Computer Vision
Verified
Journal Article
Open access
PDF file
ZYZitong Yu
Zitong Yu
YSYuming Shen
Yuming Shen
Jingang ShiJingang Shi
Jingang Shi
HZHengshuang Zhao
Hengshuang Zhao
YCYawen Cui
Yawen Cui
GZGuoying Zhao
Guoying Zhao
Remote photoplethysmography (rPPG), which aims at measuring heart activities and physiological signals from facial video without any contact, has great potential in many applications (e.g., remote healthcare and affective computing). Recent deep learning approaches focus on mining subtle rPPG clues using convolutional neural networks with limited spatio-temporal receptive fields, which neglect the long-range spatio-temporal perception and interaction for rPPG modeling. In this paper, we propose two end-to-end video transformer based architectures, namely PhysFormer and PhysFormer++, to adaptively aggregate both local and global spatio-temporal features for rPPG representation enhancement. As key modules in PhysFormer, the temporal difference transformers first enhance the quasi-periodic rPPG features with temporal difference guided global attention, and then refine the local spatio-temporal representation against interference. To better exploit the temporal contextual and periodic rPPG clues, we also extend the PhysFormer to the two-pathway SlowFast based PhysFormer++ with temporal difference periodic and cross-attention transformers. Furthermore, we propose the label distribution learning and a curriculum learning inspired dynamic constraint in frequency domain, which provide elaborate supervisions for PhysFormer and PhysFormer++ and alleviate overfitting. Comprehensive experiments are performed on four benchmark datasets to show our superior performance on both intra- and cross-dataset testings. Unlike most transformer networks needed pretraining from large-scale datasets, the proposed PhysFormer family can be easily trained from scratch on rPPG datasets, which makes it promising as a novel transformer baseline for the rPPG community.

13 January 2025 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Verified
Journal Article
Closed access
LYLihe Yang
Lihe Yang
ZZZhen Zhao
Zhen Zhao
HZHengshuang Zhao
Hengshuang Zhao
Semi-supervised semantic segmentation (SSS) aims at learning rich visual knowledge from cheap unlabeled images to enhance semantic segmentation capability. Among recent works, UniMatch (Yang et al. 2023) improves its precedents tremendously by amplifying the practice of weak-to-strong consistency regularization. Subsequent works typically follow similar pipelines and propose various delicate designs. Despite the achieved progress, strangely, even in this flourishing era of numerous powerful vision models, almost all SSS works are still sticking to 1) using outdated ResNet encoders with small-scale ImageNet-1 K pre-training, and 2) evaluation on simple Pascal and Cityscapes datasets. In this work, we argue that, it is necessary to switch the baseline of SSS from ResNet-based encoders to more capable ViT-based encoders (e.g., DINOv2) that are pre-trained on massive data. A simple update on the encoder (even using 2× fewer parameters) can bring more significant improvement than careful method designs. Built on this competitive baseline, we present our upgraded and simplified UniMatch V2, inheriting the core spirit of weak-to-strong consistency from V1, but requiring less training cost and providing consistently better results. Additionally, witnessing the gradually saturated performance on Pascal and Cityscapes, we appeal that we should focus on more challenging benchmarks with complex taxonomy, such as ADE20K and COCO datasets.

6 December 2024
Verified
Book Chapter
Closed access
PDF file
ZZZheng Zhang
Zheng Zhang
WHWenbo Hu
Wenbo Hu
YLYixing Lao
Yixing Lao
THTong He
Tong He
HZHengshuang Zhao
Hengshuang Zhao