Ming-Yu Liu

Ming-Yu Liu

Auto-generated profile, based on publicly available information

Ming-Yu Liu

NVIDIA, Santa Clara, CA, USA

Research Focus & Metrics

Academic Career Length

7Years (2020-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 February 2021 in Proceedings of the IEEE
Verified
Journal Article
Closed access
MLMing-Yu Liu
Ming-Yu Liu
XHXun Huang
Xun Huang
JYJiahui Yu
Jiahui Yu
TWTing-Chun Wang
Ting-Chun Wang
AMArun Mallya
Arun Mallya
The generative adversarial network (GAN) framework has emerged as a powerful tool for various image and video synthesis tasks, allowing the synthesis of visual content in an unconditional or input-conditional manner. It has enabled the generation of high-resolution photorealistic images and videos, a task that was challenging or impossible with prior methods. It has also led to the creation of many new applications in content creation. In this article, we provide an overview of GANs with a special focus on algorithms and applications for visual synthesis. We cover several important techniques to stabilize GAN training, which has a reputation for being notoriously difficult. We also discuss its applications to image translation, image processing, video synthesis, and neural rendering.


24 January 2020 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Verified
Journal Article
Closed access
ADAysegul Dundar
Aysegul Dundar
MLMing-Yu Liu
Ming-Yu Liu
ZYZhiding Yu
Zhiding Yu
TWTing-Chun Wang
Ting-Chun Wang
JZJohn Zedlewski
John Zedlewski
JKJan Kautz
Jan Kautz
Generating computer graphics (CG) rendered synthetic images has been widely used to create simulation environments for robotics/autonomous driving and generate labeled data. Yet, the problem of training models purely with synthetic data remains challenging due to the considerable domain gaps caused by current limitations on rendering. In this paper, we propose a simple yet effective domain adaptation framework towards closing such gap at image level. Unlike many GAN-based approaches, our method aims to match the covariance of the universal feature embeddings across domains, making the adaptation a fast, convenient "on-the-fly" step and avoiding the need for potentially difficult GAN trainings. To align domains more precisely, we further propose a conditional covariance matching framework which iteratively estimates semantic segmentation regions and conditionally matches the class-wise feature covariance given the segmentation regions. We demonstrate that both tasks can mutually refine and considerably improve each other, leading to state-of-the-art domain adaptation results. Extensive experiments under multiple synthetic-to-real settings show that our approach exceeds the performance of latest domain adaptation approaches. In addition, we offer a quantitative analysis where our framework shows considerable reduction in Frechet Inception distance between source and target domains, demonstrating the effectiveness of this work in bridging the synthetic-to-real domain gap.

28 October 2024
Verified
Conference Paper
Open access
SDShuqi Dai
Shuqi Dai
MLMing-Yu Liu
Ming-Yu Liu
RVRafael Valle
Rafael Valle
SGSiddharth Gururani
Siddharth Gururani
Singing Voice Synthesis (SVS) has significantly advanced with deep generative models, achieving high audio quality but still struggling with musicality, mainly due to the lack of performance control over timing, dynamics, and pitch, which are essential for music expression. Additionally, integrating data and supporting diverse languages and styles in SVS remain challenging. To tackle these issues, this paper presents ExpressiveSinger, an SVS framework that leverages a cascade of diffusion models to generate realistic singing across multiple languages, styles, and techniques from scores and lyrics. Our approach begins with consolidating, cleaning, annotating, and processing public singing datasets, developing a multilingual phoneme set, and incorporating different musical styles and techniques. We then design methods for generating expressive performance control signals including phoneme timing, F0 curves, and amplitude envelopes, which enhance musicality and model consistency, introduce more controllability, and reduce data requirements. Finally, we generate mel-spectrograms and audio from performance control signals with style guidance and singer timbre embedding. Our models also enable trained singers to sing in new languages and styles. Several listening tests reveal both musicality and controllability of our generated singing compared with existing works and human singing. We release the data for future research. Demo: https://shuqid.net/expressive-singing-synthesis.

25 July 2024
Verified
Conference Paper
Open access
TLTsung-Yi Lin
Tsung-Yi Lin
CLChen-Hsuan Lin
Chen-Hsuan Lin
YCYin Cui
Yin Cui
YGYunhao Ge
Yunhao Ge
SNSeungjun Nah
Seungjun Nah
MLMing-Yu Liu
Ming-Yu Liu
We introduce GenUSD, an end-to-end text-to-scene generation framework that transforms natural language queries into realistic 3D scenes, including 3D objects and layouts. The process involves two main steps: 1) A Large Language Model (LLM) generates a scene layout hierarchically. It first proposes a high-level plan to decompose the scene into multiple functionally and spatially distinct subscenes. Then, for each subscene, the LLM proposes objects with detailed positions, poses, sizes, and descriptions. To manage complex object relationships and intricate scenes, we introduce object layout design meta functions as tools for the LLM. 2) A novel text-to-3D model generates each 3D object with surface meshes and high-resolution texture maps based on the LLM’s descriptions. The assembled 3D assets form the final 3D scene, represented as a Universal Scene Description (USD) format. GenUSD ensures physical plausibility by incorporating functions to prevent collisions.