Jiayi Lyu

Jiayi Lyu

Auto-generated profile, based on publicly available information

Jiayi Lyu

School of Engineering Science, University of Chinese Academy of Sciences, Beijing, China

Research Focus & Metrics

Academic Career Length

7Years (2020-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
3 November 2023 in IEEE Transactions on Multimedia
Verified
Journal Article
Closed access
XLXing Lan
Xing Lan
JLJiayi Lyu
Jiayi Lyu
HJHanyu Jiang
Hanyu Jiang
KDKun Dong
Kun Dong
ZNZehai Niu
Zehai Niu
JXJian Xue
Jian Xue
In this paper, we explore the zero-shot capability of the Segment Anything Model (SAM) for food image segmentation. To address the lack of class-specific information in SAM-generated masks, we propose a novel framework, called FoodSAM . This innovative approach integrates the coarse semantic mask with SAM-generated masks to enhance semantic segmentation quality. Besides, we recognize that the ingredients in food can be supposed as independent individuals, which motivated us to perform instance segmentation on food images. Furthermore, FoodSAM extends its zero-shot capability to encompass panoptic segmentation by incorporating an object detector, which renders FoodSAM to effectively capture non-food object information. Drawing inspiration from the recent success of promptable segmentation, we also extend FoodSAM to promptable segmentation, supporting various prompt variants. Consequently, FoodSAM emerges as an all-encompassing solution capable of segmenting food items at multiple levels of granularity. Remarkably, this pioneering framework stands as the first-ever work to achieve instance, panoptic, and promptable segmentation on food images. Extensive experiments demonstrate the feasibility and impressing performance of FoodSAM, validating SAM's potential as a prominent and influential tool within the domain of food image segmentation.

21 October 2023 in ACM Computing Surveys
Verified
Journal Article
Open access
GXGuoyang Xie
Guoyang Xie
YHYawen Huang
Yawen Huang
Jinbao WangJinbao Wang
Jinbao Wang
JLJiayi Lyu
Jiayi Lyu
FZFeng Zheng
Feng Zheng
YJYaochu Jin
Yaochu Jin
Multi-modality imaging improves disease diagnosis and reveals distinct deviations in tissues with anatomical properties. The existence of completely aligned and paired multi-modality neuroimaging data has proved its effectiveness in brain research. However, collecting fully aligned and paired data is expensive or even impractical, since it faces many difficulties, including high cost, long acquisition time, image corruption, and privacy issues. An alternative solution is to explore unsupervised or weakly supervised learning methods to synthesize the absent neuroimaging data. In this article, we provide a comprehensive review of cross-modality synthesis for neuroimages, from the perspectives of weakly supervised and unsupervised settings, loss functions, evaluation metrics, imaging modalities, datasets, and downstream applications based on synthesis. We begin by highlighting several opening challenges for cross-modality neuroimage synthesis. Then, we discuss representative architectures of cross-modality synthesis methods under different supervisions. This is followed by a stepwise in-depth analysis to evaluate how cross-modality neuroimage synthesis improves the performance of its downstream tasks. Finally, we summarize the existing research findings and point out future research directions. All resources are available at https://github.com/M-3LAB/awesome-multimodal-brain-image-systhesis.

11 July 2023 in Complex & Intelligent Systems
Verified
Journal Article
Open access
PDF file
GXGuoyang Xie
Guoyang Xie
Jinbao WangJinbao Wang
Jinbao Wang
YGYu Guo
Yu Guo
JLJiayi Lyu
Jiayi Lyu
FZFeng Zheng
Feng Zheng
YJYaochu Jin
Yaochu Jin
The widely employed tiny neural networks (TNNs) in mobile devices are vulnerable to adversarial attacks. However, more advanced research on the robustness of TNNs is highly in demand. This work focuses on improving the robustness of TNNs without sacrificing the model’s accuracy. To find the optimal trade-off networks in terms of the adversarial accuracy, clean accuracy, and model size, we present TAM-NAS, a tiny adversarial multi-objective one-shot network architecture search method. First, we build a novel search space comprised of new tiny blocks and channels to establish a balance between the model size and adversarial performance. Then, we demonstrate how the supernet facilitates the acquisition of the optimal subnet under white-box adversarial attacks, provided that the supernet significantly impacts the subnet’s performance. Concretely, we investigate a new adversarial training paradigm by evaluating the adversarial transferability, the width of the supernet, and the distinction between training subnets from scratch and fine-tuning. Finally, we undertake statistical analysis for the layer-wise combination of specific blocks and channels on the first non-dominated front, which can be utilized as a design guideline for the design of TNNs.

26 December 2024 in IEEE Transactions on Circuits and Systems for Video Technology
Verified
Journal Article
Closed access
JLJiayi Lyu
Jiayi Lyu
XLXing Lan
Xing Lan
GHGuohong Hu
Guohong Hu
HJHanyu Jiang
Hanyu Jiang
WGWei Gan
Wei Gan
JXJian Xue
Jian Xue
Talking face generation focuses on creating natural facial animations that align with the provided text or audio input. Current methods in this field primarily rely on facial landmarks to convey emotional changes. However, spatial key-points are valuable, yet limited in capturing the intricate dynamics and subtle nuances of emotional expressions due to their restricted spatial coverage. Consequently, this reliance on sparse landmarks can result in decreased accuracy and visual quality, especially when representing complex emotional states. To address this issue, we propose a novel method called Emotional Talking with Action Unit (ETAU), which seamlessly integrates facial Action Units (AUs) into the generation process. Unlike previous works that solely rely on facial landmarks, ETAU employs both Action Units and landmarks to comprehensively represent facial expressions through interpretable representations. Our method provides a detailed and dynamic representation of emotions by capturing the complex interactions among facial muscle movements. Moreover, ETAU adopts a multi-modal strategy by seamlessly integrating emotion prompts, driving videos, and target images, and by leveraging various input data effectively, it generates highly realistic and emotional talking-face videos. Through extensive evaluations across multiple datasets, including MEAD, LRW, GRID and HDTF, ETAU outperforms previous methods, showcasing its superior ability to generate high-quality, expressive talking faces with improved visual fidelity and synchronization. Moreover, ETAU exhibits a significant improvement on the emotion accuracy of the generated results, reaching an impressive average accuracy of 84% on the MEAD dataset.

13 August 2024 in IEEE Transactions on Circuits and Systems for Video Technology
Verified
Journal Article
Closed access
XLXing Lan
Xing Lan
JLJiayi Lyu
Jiayi Lyu
KDKun Dong
Kun Dong
HJHanyu Jiang
Hanyu Jiang
庆胡庆浩 胡
庆浩 胡
JXJian Xue
Jian Xue
Heatmap-based methods have dominated the face alignment task, yet the maximum response decoding scheme necessitates further reform. While some studies have attempted to compensate for prediction offsets using a post-processing module, the prediction errors induced by the maximum response decoding scheme remain challenging to rectify. In this paper, we assume that using heatmap value to denote the ground-truth probability is not accurate enough. To cure this problem, we propose DISPAL, a novel DIS tribution-based P robability for f A cial L andmarks, which signifies the ground-truth probability by the similarity between the pixel’s neighbouring value distribution and Gaussian distribution. This innovative probability enables us to pinpoint the keypoint location more robustly than previous methods that rely solely on the peak score. It also exhibits remarkable generalization to complex decoding methodologies. Furthermore, we propose supervising this probability as an additional task loss to help the model learn better heatmap representation. Extensive empirical results on WFLW, 300W, and COFW datasets demonstrate that our distribution-based probability mechanism significantly surpasses original value-based probability approaches.

1 January 2020 in Journal of Physics: Conference Series
Verified
Conference Paper
Open access
JLJiayi Lyu
Jiayi Lyu
NHNing He
Ning He
XSXin Sun
Xin Sun
JXJian Xue
Jian Xue
Traditional correlation filters have high tracking efficiency, but because of the negative samples in the correlation filter that are obtained by moving the object patch itself, the real background information is not included in the sample composition. In this paper, the method for obtaining and detecting the background spatial information in multiple channel is given. A method that combines the background spatial information is integrated into the framework of correlation filter, and an experiment was carried out on existing publicly available datasets. The experimental results show that the tracking effect in some complicated scenes (e.g., illumination changes, fast motion, scale changes, and occlusion) is improved. This approach can effectively improve the success rate and accuracy of object tracking in complicated scenes without affecting the frame rate, obtaining a performance that is superior to that of other correlation filters.