Jiayi Guo

Jiayi Guo

Auto-generated profile, based on publicly available information

Jiayi Guo

Department of Automation, BNRist, Tsinghua University, Beijing, China

Research Focus & Metrics

Academic Career Length

6Years (2021-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
22 November 2024
Verified
Book Chapter
Closed access
PDF file
YPYifan Pu
Yifan Pu
ZXZhuofan Xia
Zhuofan Xia
JGJiayi Guo
Jiayi Guo
DHDongchen Han
Dongchen Han
QLQixiu Li
Qixiu Li
XLXiu Li
Xiu Li

7 November 2023 in IEEE Transactions on Circuits and Systems for Video Technology
Verified
Journal Article
Closed access
JGJiayi Guo
Jiayi Guo
HMHayk Manukyan
Hayk Manukyan
CYChenyu Yang
Chenyu Yang
CWChaofei Wang
Chaofei Wang
LKLevon Khachatryan
Levon Khachatryan
GHGao Huang
Gao Huang
The existing image-to-video translation methods generally follow a frame-by-frame generative paradigm, while extracting the temporal information from a reference video or an audio stream. Inspired by the recent success in text-guided image generation, we explore a more challenging but promising task, Text-guided Image-to-Video (TI2V) translation. Given an image and a brief text description as input, TI2V aims to generate a facial expression video following the image and text. To this end, we first propose an automatic video captioning pipeline to generate dense textual descriptions for facial video datasets, using both expression labels and action units. These dense textual descriptions provide precise semantic guidance for TI2V learning. Then we design and train an efficient framework, FaceCLIP, on these datasets to deal with the TI2V translation task. FaceCLIP adopts a video autoencoder to model the temporal information of training videos, and a pretrained CLIP model to embed the video frames and the text description. We design a reconstruction loss and an embedding alignment loss to train the autoencoder to obtain the text-guided video generative ability. Recognizing that expressions are closely tied to facial landmark motions, the reconstruction loss is applied to facial landmarks rather than each video frame, significantly enhancing training efficiency. We compare FaceCLIP with several potential baseline methods, and extensively evaluate the performance using multiple metrics. Both qualitative and quantitative results validate the superiority of FaceCLIP in terms of both visual quality and expression-text consistency. Moreover, the unique ability of FaceCLIP to generate videos based on abstract texts demonstrates its stronger generalization capability.

29 October 2024
Verified
Book Chapter
Closed access
PDF file
ZNZanlin Ni
Zanlin Ni
YWYulin Wang
Yulin Wang
RZRenping Zhou
Renping Zhou
RLRui Lu
Rui Lu
JGJiayi Guo
Jiayi Guo
GHGao Huang
Gao Huang

31 October 2025 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Verified
Journal Article
Closed access
ZNZanlin Ni
Zanlin Ni
YWYulin Wang
Yulin Wang
YHYeguo Hua
Yeguo Hua
RZRenping Zhou
Renping Zhou
JGJiayi Guo
Jiayi Guo
GHGao Huang
Gao Huang
Recent advances in image synthesis have been propelled by powerful generative models, such as Masked Generative Transformers (MaskGIT), autoregressive models, diffusion models, and rectified flow models. A common principle behind their success is the decomposition of complex synthesis tasks into multiple tractable steps. However, this introduces a proliferation of step-specific parameters to be configured for modulating the iterative generation process (e.g., mask ratio, noise level, or temperature at each step). Existing approaches typically rely on manually-designed scheduling rules to manage this complexity, demanding expert knowledge and extensive trial-and-error. Furthermore, these static schedules lack the flexibility to adapt to the unique characteristics of each individual sample, yielding sub-optimal performance. To address this issue, we present AdaGen, a general, learnable, and sample-adaptive framework for scheduling the iterative generation process. Specifically, we formulate the scheduling problem as a Markov Decision Process, where a lightweight policy network is introduced to adaptively determine the most suitable parameters given the current generation state, and can be trained through reinforcement learning. Importantly, we demonstrate that simple reward designs, such as FID or pre-trained reward models, can be easily hacked and may not reliably guarantee the desired quality or diversity of generated samples. Therefore, we propose an adversarial reward design to guide the training of the policy networks effectively. Finally, we introduce an inference-time refinement strategy and a controllable fidelity-diversity trade-off mechanism to further enhance the performance and flexibility of AdaGen. Comprehensive experiments across five benchmark datasets (ImageNet-256 × 256 & 512 × 512, MS-COCO, CC3M, and LAION-5B) and four distinct generative paradigms validate the superiority of AdaGen . For example, AdaGen achieves better performance on DiT-XL with $\mathbf {\sim 3\times }$ lower inference cost and improves the FID of VAR from 1.92 to 1.59 with negligible additional computational overhead.

6 May 2021 in arXiv
Verified
Preprint
Open access
PDF file
ALAng Li
Ang Li
JGJiayi Guo
Jiayi Guo
HYHuanrui Yang
Huanrui Yang
FSFlora D. Salim
Flora D. Salim
YCYiran Chen
Yiran Chen
Deep learning has been widely applied in many computer vision applications, with remarkable success. However, running deep learning models on mobile devices is generally challenging due to the limitation of computing resources. A popular alternative is to use cloud services to run deep learning models to process raw data. This, however, imposes privacy risks. Some prior arts proposed sending the features extracted from raw data to the cloud. Unfortunately, these extracted features can still be exploited by attackers to recover raw images and to infer embedded private attributes. In this paper, we propose an adversarial training framework, DeepObfuscator, which prevents the usage of the features for reconstruction of the raw images and inference of private attributes. This is done while retaining useful information for the intended cloud service. DeepObfuscator includes a learnable obfuscator that is designed to hide privacy-related sensitive information from the features by performing our proposed adversarial training algorithm. The proposed algorithm is designed by simulating the game between an attacker who makes efforts to reconstruct raw image and infer private attributes from the extracted features and a defender who aims to protect user privacy. By deploying the trained obfuscator on the smartphone, features can be locally extracted and then sent to the cloud. Our experiments on CelebA and LFW datasets show that the quality of the reconstructed images from the obfuscated features of the raw image is dramatically decreased from 0.9458 to 0.3175 in terms of multi-scale structural similarity. The person in the reconstructed image, hence, becomes hardly to be re-identified. The classification accuracy of the inferred private attributes that can be achieved by the attacker is significantly reduced to a random-guessing level.

11 August 2024 in arXiv
Verified
Preprint
Open access
PDF file
YPYifan Pu
Yifan Pu
ZXZhuofan Xia
Zhuofan Xia
JGJiayi Guo
Jiayi Guo
DHDongchen Han
Dongchen Han
QLQixiu Li
Qixiu Li
XLXiu Li
Xiu Li
This paper identifies significant redundancy in the query-key interactions within self-attention mechanisms of diffusion transformer models, particularly during the early stages of denoising diffusion steps. In response to this observation, we present a novel diffusion transformer framework incorporating an additional set of mediator tokens to engage with queries and keys separately. By modulating the number of mediator tokens during the denoising generation phases, our model initiates the denoising process with a precise, non-ambiguous stage and gradually transitions to a phase enriched with detail. Concurrently, integrating mediator tokens simplifies the attention module's complexity to a linear scale, enhancing the efficiency of global attention processes. Additionally, we propose a time-step dynamic mediator token adjustment mechanism that further decreases the required computational FLOPs for generation, simultaneously facilitating the generation of high-quality images within the constraints of varied inference budgets. Extensive experiments demonstrate that the proposed method can improve the generated image quality while also reducing the inference cost of diffusion transformers. When integrated with the recent work SiT, our method achieves a state-of-the-art FID score of 2.01. The source code is available at https://github.com/LeapLabTHU/Attention-Mediators.