Hang Zhao

Hang Zhao

orcid

Research Focus & Metrics

Academic Career Length

9Years (2018-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
6 October 2018
Verified
Book Chapter
Open access
PDF file
HZHang Zhao
Hang Zhao
CGChuang Gan
Chuang Gan
ARAndrew Rouditchenko
Andrew Rouditchenko
CVCarl Vondrick
Carl Vondrick
JMJosh McDermott
Josh McDermott
ATAntonio Torralba
Antonio Torralba
We introduce PixelPlayer, a system that, by leveraging large amounts of unlabeled videos, learns to locate image regions which produce sounds and separate the input sounds into a set of components that represents the sound from each pixel. Our approach capitalizes on the natural synchronization of the visual and audio modalities to learn models that jointly parse sounds and images, without requiring additional manual supervision. Experimental results on a newly collected MUSIC dataset show that our proposed Mix-and-Separate framework outperforms several baselines on source separation. Qualitative results suggest our model learns to ground sounds in vision, enabling applications such as independently adjusting the volume of sound sources.


27 September 2023 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Verified
Journal Article
Open access
RZrunning zhao
running zhao
JYJiangtao Yu
Jiangtao Yu
HZHang Zhao
Hang Zhao
ENEdith Ngai
Edith Ngai
Millimeter wave (mmWave) based speech recognition provides more possibility for audio-related applications, such as conference speech transcription and eavesdropping. However, considering the practicality in real scenarios, latency and recognizable vocabulary size are two critical factors that cannot be overlooked. In this paper, we propose Radio2Text, the first mmWave-based system for streaming automatic speech recognition (ASR) with a vocabulary size exceeding 13,000 words. Radio2Text is based on a tailored streaming Transformer that is capable of effectively learning representations of speech-related features, paving the way for streaming ASR with a large vocabulary. To alleviate the deficiency of streaming networks unable to access entire future inputs, we propose the Guidance Initialization that facilitates the transfer of feature knowledge related to the global context from the non-streaming Transformer to the tailored streaming Transformer through weight inheritance. Further, we propose a cross-modal structure based on knowledge distillation (KD), named cross-modal KD, to mitigate the negative effect of low quality mmWave signals on recognition performance. In the cross-modal KD, the audio streaming Transformer provides feature and response guidance that inherit fruitful and accurate speech information to supervise the training of the tailored radio streaming Transformer. The experimental results show that our Radio2Text can achieve a character error rate of 5.7% and a word error rate of 9.4% for the recognition of a vocabulary consisting of over 13,000 words.

29 October 2024
Verified
Book Chapter
Closed access
PDF file
TYTianyuan Yuan
Tianyuan Yuan
YMYucheng Mao
Yucheng Mao
JYJiawei Yang
Jiawei Yang
YLYicheng Liu
Yicheng Liu
YWYue Wang
Yue Wang
HZHang Zhao
Hang Zhao

4 December 2024
Verified
Book Chapter
Closed access
PDF file
ZYZhangchen Ye
Zhangchen Ye
TJTao Jiang
Tao Jiang
CXChenfeng Xu
Chenfeng Xu
YLYiming Li
Yiming Li
HZHang Zhao
Hang Zhao

9 November 2023 in arXiv
Verified
Preprint
Open access
PDF file
SLSimian Luo
Simian Luo
YTYiqin Tan
Yiqin Tan
SPSuraj Patil
Suraj Patil
DGDaniel Gu
Daniel Gu
PPPatrick von Platen
Patrick von Platen
HZHang Zhao
Hang Zhao
Latent Consistency Models (LCMs) have achieved impressive performance in accelerating text-to-image generative tasks, producing high-quality images with minimal inference steps. LCMs are distilled from pre-trained latent diffusion models (LDMs), requiring only ~32 A100 GPU training hours. This report further extends LCMs' potential in two aspects: First, by applying LoRA distillation to Stable-Diffusion models including SD-V1.5, SSD-1B, and SDXL, we have expanded LCM's scope to larger models with significantly less memory consumption, achieving superior image generation quality. Second, we identify the LoRA parameters obtained through LCM distillation as a universal Stable-Diffusion acceleration module, named LCM-LoRA. LCM-LoRA can be directly plugged into various Stable-Diffusion fine-tuned models or LoRAs without training, thus representing a universally applicable accelerator for diverse image generation tasks. Compared with previous numerical PF-ODE solvers such as DDIM, DPM-Solver, LCM-LoRA can be viewed as a plug-in neural PF-ODE solver that possesses strong generalization abilities. Project page: https://github.com/luosiallen/latent-consistency-model.