Haiyu Zhao

Haiyu Zhao

Auto-generated profile, based on publicly available information

Haiyu Zhao

SenseTime Research

Research Focus & Metrics

Academic Career Length

10Years (2017-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
27 August 2024 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Verified
Journal Article
Closed access
ZCZhongang Cai
Zhongang Cai
MZMingyuan Zhang
Mingyuan Zhang
JRJiawei Ren
Jiawei Ren
CWChen Wei
Chen Wei
DRDaxuan Ren
Daxuan Ren
ZLZiwei Liu
Ziwei Liu
Image- and video-based 3D human recovery ( i.e. , pose and shape estimation) have achieved substantial progress. However, due to the prohibitive cost of motion capture, existing datasets are often limited in scale and diversity. In this work, we obtain massive human sequences by playing the video game with automatically annotated 3D ground truths. Specifically, we contribute GTA-Human, a large-scale 3D human dataset generated with the GTA-V game engine, featuring a highly diverse set of subjects, actions, and scenarios. More importantly, we study the use of game-playing data and obtain five major insights. First , game-playing data is surprisingly effective. A simple frame-based baseline trained on GTA-Human outperforms more sophisticated methods by a large margin. For videobased methods, GTA-Human is even on par with the in-domain training set. Second , we discover that synthetic data provides critical complements to the real data that is typically collected indoor. We highlight that our investigation into domain gap provides explanations for our data mixture strategies that are simple yet useful, which offers new insights to the research community. Third , the scale of the dataset matters. The performance boost is closely related to the additional data available. A systematic study on multiple key factors (such as camera angle and body pose) reveals that the model performance is sensitive to data density. Fourth , the effectiveness of GTA-Human is also attributed to the rich collection of strong supervision labels (SMPL parameters), which are otherwise expensive to acquire in real datasets. Fifth , the benefits of synthetic data extend to larger models such as deeper convolutional neural networks (CNNs) and Transformers, for which a significant impact is also observed. We hope our work could pave the way for scaling up 3D human recovery to the real world. Homepage: https://caizhongang.github.io/projects/GTA-Human/ .

21 April 2023 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Verified
Journal Article
Closed access
LPLiang Pan
Liang Pan
XCXinyi Chen
Xinyi Chen
ZCZhongang Cai
Zhongang Cai
JZJunzhe Zhang
Junzhe Zhang
HZHaiyu Zhao
Haiyu Zhao
ZLZiwei Liu
Ziwei Liu
Real-scanned point clouds are often incomplete due to viewpoint, occlusion, and noise, which hampers 3D geometric modeling and perception. Existing point cloud completion methods tend to generate global shape skeletons and hence lack fine local details. Furthermore, they mostly learn a deterministic partial-to-complete mapping, but overlook structural relations in man-made objects. To tackle these challenges, this paper proposes a variational framework, V ariational R elational point C ompletion network (VRCNet) with two appealing properties: 1) Probabilistic Modeling. In particular, we propose a dual-path architecture to enable principled probabilistic modeling across partial and complete clouds. One path consumes complete point clouds for reconstruction by learning a point VAE. The other path generates complete shapes for partial point clouds, whose embedded distribution is guided by distribution obtained from the reconstruction path during training. 2) Relational Enhancement. Specifically, we carefully design point self-attention kernel and point selective kernel module to exploit relational point features, which refines local shape details conditioned on the coarse completion. In addition, we contribute multi-view partial point cloud datasets (MVP and MVP-40 dataset) containing over 200,000 high-quality scans, which render partial 3D shapes from 26 uniformly distributed camera poses for each 3D CAD model. Extensive experiments demonstrate that VRCNet outperforms state-of-the-art methods on all standard point cloud completion benchmarks. Notably, VRCNet shows great generalizability and robustness on real-world point cloud scans. Moreover, we can achieve robust 3D classification for partial point clouds with the help of VRCNet, which can highly increase classification accuracy.

5 March 2024 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Verified
Journal Article
Closed access
ZLZhipeng Luo
Zhipeng Luo
CZChangqing Zhou
Changqing Zhou
LPLiang Pan
Liang Pan
GZGongjie Zhang
Gongjie Zhang
TLTianrui Liu
Tianrui Liu
SLShijian Lu
Shijian Lu
With the prevalent use of LiDAR sensors in autonomous driving, 3D point cloud object tracking has received increasing attention. In a point cloud sequence, 3D object tracking aims to predict the location and orientation of an object in consecutive frames. Motivated by the success of transformers, we propose P oint T racking TR ansformer (PTTR), which efficiently predicts high-quality 3D tracking results in a coarse-to-fine manner with the help of transformer operations. PTTR consists of three novel designs. 1) Instead of random sampling, we design Relation-Aware Sampling to preserve relevant points to the given template during subsampling. 2) We propose a Point Relation Transformer for effective feature aggregation and feature matching between the template and search region. 3) Based on the coarse tracking results, we employ a novel Prediction Refinement Module to obtain the final refined prediction through local feature pooling. In addition, motivated by the favorable properties of the Bird's-Eye View (BEV) of point clouds in capturing object motion, we further design a more advanced framework named PTTR++, which incorporates both the point-wise view and BEV representation to exploit their complementary effect in generating high-quality tracking results. PTTR++ substantially boosts the tracking performance on top of PTTR with low computational overhead. Extensive experiments over multiple datasets show that our proposed approaches achieve superior 3D tracking accuracy and efficiency. Code will be available at https://github.com/Jasonkks/PTTR


19 January 2024 in arXiv
Verified
Preprint
Open access
PDF file
ZSZhuoran Shen
Zhuoran Shen
MZMingyuan Zhang
Mingyuan Zhang
HZHaiyu Zhao
Haiyu Zhao
SYShuai Yi
Shuai Yi
HLHongsheng Li
Hongsheng Li
Dot-product attention has wide applications in computer vision and natural language processing. However, its memory and computational costs grow quadratically with the input size. Such growth prohibits its application on high-resolution inputs. To remedy this drawback, this paper proposes a novel efficient attention mechanism equivalent to dot-product attention but with substantially less memory and computational costs. Its resource efficiency allows more widespread and flexible integration of attention modules into a network, which leads to better accuracies. Empirical evaluations demonstrated the effectiveness of its advantages. Efficient attention modules brought significant performance boosts to object detectors and instance segmenters on MS-COCO 2017. Further, the resource efficiency democratizes attention to complex models, where high costs prohibit the use of dot-product attention. As an exemplar, a model with efficient attention achieved state-of-the-art accuracies for stereo depth estimation on the Scene Flow dataset. Code is available at https://github.com/cmsflash/efficient-attention.

15 November 2022 in arXiv
Verified
Preprint
Open access
PDF file
KCKennard Yanting Chan
Kennard Yanting Chan
GLGuosheng Lin
Guosheng Lin
HZHaiyu Zhao
Haiyu Zhao
WLWeisi Lin
Weisi Lin
We propose IntegratedPIFu, a new pixel aligned implicit model that builds on the foundation set by PIFuHD. IntegratedPIFu shows how depth and human parsing information can be predicted and capitalised upon in a pixel-aligned implicit model. In addition, IntegratedPIFu introduces depth oriented sampling, a novel training scheme that improve any pixel aligned implicit model ability to reconstruct important human features without noisy artefacts. Lastly, IntegratedPIFu presents a new architecture that, despite using less model parameters than PIFuHD, is able to improves the structural correctness of reconstructed meshes. Our results show that IntegratedPIFu significantly outperforms existing state of the arts methods on single view human reconstruction. Our code has been made available online.