Cong Fan

Cong Fan

Auto-generated profile, based on publicly available information

Cong Fan

Xinjiang University

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
9 April 2023 in Electronics
Verified
Journal Article
Open access
PDF file
JLJiajun Liu
Jiajun Liu
AWAishan Wumaier
Aishan Wumaier
CFCong Fan
Cong Fan
SGshen Mr. guo
shen Mr. guo
The automatic fluency assessment of spontaneous speech without reference text is a challenging task that heavily depends on the accuracy of automatic speech recognition (ASR). Considering this scenario, it is necessary to explore an assessment method that combines ASR. This is mainly due to the fact that in addition to acoustic features being essential for assessment, the text features output by ASR may also contain potentially fluency information. However, most existing studies on automatic fluency assessment of spontaneous speech are based solely on audio features, without utilizing textual information, which may lead to a limited understanding of fluency features. To address this, we propose a multimodal automatic speech fluency assessment method that combines ASR output. Specifically, we first explore the relevance of the fluency assessment task to the ASR task and fine-tune the Wav2Vec2.0 model using multi-task learning to jointly optimize the ASR task and fluency assessment task, resulting in both the fluency assessment results and the ASR output. Then, the text features and audio features obtained from the fine-tuned model are fed into the multimodal fluency assessment model, using attention mechanisms to obtain more reliable assessment results. Finally, experiments on the PSCPSF and Speechocean762 dataset suggest that our proposed method performs well in different assessment scenarios.

22 May 2023 in IEEE Access
Verified
Journal Article
Open access
SGshen Mr. guo
shen Mr. guo
ZKZaokere Kadeer
Zaokere Kadeer
AWAishan Wumaier
Aishan Wumaier
LWLiejun Wang
Liejun Wang
CFCong Fan
Cong Fan
In recent years, with the development of deep learning, research on end-to-end mispronunciation detection and diagnosis(MDD) methods has been further promoted. At present, research on end-to-end mispronunciation detection and diagnosis is gradually emerging. Most end-to-end mispronunciation detection and diagnosis methods are based on the CNN-RNN-CTC network structure. To improve the performance of end-to-end mispronunciation detection and diagnosis systems, this paper proposes an end-to-end multi-feature and multi-modal mispronunciation detection and diagnosis method based on the Squeezeformer encoder. The model uses Squeezeformer as an audio encoder, a Bi-LSTM network as a phoneme encoder, and Transformer as a decoder. The model fuses phoneme information before speech encoding and decoding, respectively, and uses a secondary decoding mechanism during the decoding process. This study further incorporated phoneme information in the encoding process so that the model could learn the intrinsic characteristics of the speaker’s pronunciation content. The decoding process uses a secondary decoding mechanism to send the sequence decoded by the model to the decoder for decoding again, which solves the problem of no a priori knowledge at the decoder end in the first decoding stage, thus improving the performance of mispronunciation detection and diagnosis. In this study, experiments were conducted on the PSC-Reading Mandarin mispronunciation detection and diagnosis dataset. Compared with the baseline model, the F1 index improved from 0.4060 to 0.7943, and the diagnostic accuracy improved from 83.93% to 88.45%.