Yu Pan

Yu Pan

Auto-generated profile, based on publicly available information

Yu Pan

Kyushu University

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
31 March 2025 in IEEE Transactions on Audio, Speech and Language Processing
Verified
Journal Article
Closed access
JYJixun Yao
Jixun Yao
QWQing Wang
Qing Wang
PGPengcheng Guo
Pengcheng Guo
ZNZiqian Ning
Ziqian Ning
YYYuguang Yang
Yuguang Yang
LXLei Xie
Lei Xie
Speaker anonymization is an effective privacy protection solution designed to conceal the speaker's identity while preserving the linguistic content and para-linguistic information of the original speech. While most prior studies focus solely on a single language, an ideal speaker anonymization system should be capable of handling multiple languages. This paper proposes MUSA, a MUlti-lingual Speaker Anonymization approach that employs a serial disentanglement strategy to perform a step-by-step disentanglement from a global time-invariant representation to a temporal time-variant representation. By utilizing semantic distillation and self-supervised speaker distillation, the serial disentanglement strategy can avoid strong inductive biases and exhibit superior generalization performance across different languages. Meanwhile, we propose a straightforward anonymization strategy that employs empty embedding with zero values to simulate the speaker identity concealment process, eliminating the need for conversion to a pseudo-speaker identity and thereby reducing the complexity of speaker anonymization process. Experimental results on VoicePrivacy official datasets and multi-lingual datasets demonstrate that MUSA can effectively protect speaker privacy while preserving linguistic content and para-linguistic information.

26 December 2025 in IEEE Transactions on Audio, Speech and Language Processing
Verified
Journal Article
Closed access
JYJixun Yao
Jixun Yao
YYYang Yuguang
Yang Yuguang
YFYuan Feng
Yuan Feng
YPYu Pan
Yu Pan
ZNZiqian Ning
Ziqian Ning
LXLei Xie
Lei Xie
Integrating reinforcement learning to align generated speech with human preferences has proven effective in improving the robustness of language model-based text-to-speech (TTS) systems. Current approaches primarily rely on preference data annotated at the utterance level. However, frequent issues affecting the listening experience, such as unnatural prosody, mispronunciations, or semantic inconsistencies, often arise only in specific segments of audio, while other segments may be well-generated and require no correction. This mismatch between coarse-grained supervision and fine-grained quality variation limits the effectiveness of preference-based optimization. In this study, we propose a fine-grained preference optimization approach (FPO) to enhance the robustness of TTS systems. FPO shifts the optimization paradigm from global utterance-level tuning to targeted local refinement, focusing on addressing localized issues in generated samples rather than uniformly optimizing the entire utterance. We begin by analyzing the types of common generation issues and categorizing them into temporal modeling errors and semantic-phonetic alignment errors, which frequently degrade intelligibility and naturalness. To tackle these problems, we introduce a selective training loss strategy that leverages fine-grained labels for each issue type, allowing the model to focus on learning signals where they are most needed. Experimental results demonstrate that FPO substantially improves the robustness of zero-shot TTS systems by effectively correcting problematic regions in the output. This leads to a significant reduction in the bad case ratio, improved intelligibility, and overall perceptual quality. Moreover, FPO exhibits strong data efficiency, achieving comparable or superior performance to baseline methods while requiring notably fewer training samples. Audio samples can be found at https://yaoxunji.github.io/fpo/.

29 October 2023
Verified
Conference Paper
Open access
GYGuofeng Yi
Guofeng Yi
YYYuguang Yang
Yuguang Yang
YPYu Pan
Yu Pan
YCYuhang Cao
Yuhang Cao
JYJixun Yao
Jixun Yao
HLHeng Lu
Heng Lu

7 October 2025 in IEEE Signal Processing Letters
Verified
Journal Article
Closed access
YPYu Pan
Yu Pan
YYYuguang Yang
Yuguang Yang
JYJixun Yao
Jixun Yao
LMLei Ma
Lei Ma
JZJianjun Zhao
Jianjun Zhao
Despite recent advances in zero-shot voice conversion (VC), achieving speaker similarity and naturalness comparable to ground-truth recordings remains a significant challenge. In this letter, we propose CTEFM-VC, a zero-shot VC framework that integrates Content-aware Timbre Ensemble modeling with conditional Flow Matching. Specifically, CTEFM-VC decouples utterances into content and timbre representations and leverages a conditional flow matching model to reconstruct the Mel-spectrogram of the source speech. To enhance its timbre modeling capability and naturalness of generated speech, we first introduce a context-aware timbre ensemble modeling approach that adaptively integrates diverse speaker verification embeddings and enables the effective utilization of source content and target timbre elements through a cross-attention module. Furthermore, a structural similarity-based timbre loss is presented to jointly train CTEFM-VC end-to-end. Experiments show that CTEFM-VC consistently achieves the best performance in all metrics assessing speaker similarity, speech naturalness, and intelligibility, significantly outperforming state-of-the-art zero-shot VC systems.

24 July 2025 in TechRxiv
Verified
Preprint
Open access
YPYu Pan
Yu Pan
YYYuguang Yang
Yuguang Yang
YHYanni Hu
Yanni Hu
JYJianhao Ye
Jianhao Ye
XZXiang Zhang
Xiang Zhang
JZJianjun Zhao
Jianjun Zhao
Despite recent advances in multilingual speech-to-speech translation (S2ST), several critical challenges persist: 1) achieving high-quality translation remains a major hurdle, and 2) most existing methods heavily rely on large-scale parallel speech corpora, which are costly and difficult to obtain. To address these issues, we propose \textit{S2ST-Omni}, an efficient and scalable framework for multilingual S2ST. Specifically, we decompose the S2ST task into speech-to-text translation (S2TT) and text-to-speech synthesis (TTS). For S2TT, we propose an effective speech language model that integrates the pretrained Whisper encoder for robust audio understanding and Qwen 3.0 for advanced text comprehension. A lightweight speech adapter is employed to bridge the modality gap between speech and text representations. To further facilitate the multimodal knowledge learning, a two-stage fine-tuning strategy is introduced. In the TTS stage, we adopt a streaming autoregressive generation approach to produce natural and fluent target speech. Experiments on the CVSS benchmark show that S2ST-Omni consistently outperforms existing state-of-the-art S2ST systems in translation quality, highlighting its effectiveness and superiority.