Jiajun He

Jiajun He

Auto-generated profile, based on publicly available information

Jiajun He

Graduate School of Informatics, Nagoya University, Nagoya, Japan

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
29 October 2023
Verified
Conference Paper
Open access
JTJingguang Tian
Jingguang Tian
DHDesheng Hu
Desheng Hu
XSXiaohan Shi
Xiaohan Shi
JHJiajun He
Jiajun He
XLXingfeng Li
Xingfeng Li
XHXinhui Hu
Xinhui Hu
Multimodal emotion recognition is the task of identifying and understanding emotions by integrating information from multiple modalities, such as audio, visual, and textual data. However, the scarcity of labeled data poses a significant challenge for this task. To this end, this paper proposes a novel approach via a semi-supervised learning framework by incorporating consensus decision-making and label correction methods. Firstly, we employ supervised learning on the trimodal input data to establish robust initial models. Secondly, we generate reliable pseudo-labels for unlabelled data by leveraging consensus decision-making and label correction methods. Thirdly, we train the model in a supervised manner using both labeled and pseudo-labeled data. Moreover, the process of generating pseudo-labels and semi-supervised learning can be iterated to refine the model further. Experimental results on the MER 2023 dataset show the effectiveness of our proposed framework, achieving significant improvement on the MER-MULTI, MER-NOISE, and MER-SEMI subsets, respectively.

25 September 2025 in IEEE Transactions on Audio, Speech and Language Processing
Verified
Journal Article
Open access
JHJiajun He
Jiajun He
XSXiaohan Shi
Xiaohan Shi
CHCheng-Hung Hu
Cheng-Hung Hu
JMJinyi Mi
Jinyi Mi
XLXingfeng Li
Xingfeng Li
TTTomoki Toda
Tomoki Toda
Multimodal speech emotion recognition (SER) has emerged as pivotal for improving human–machine interaction. Researchers are increasingly leveraging both speech and textual information obtained through automatic speech recognition (ASR) to comprehensively recognize emotional states from speakers. Although this approach reduces reliance on human-annotated text data, ASR errors possibly degrade emotion recognition performance. To address this challenge, in our previous work, we introduced two auxiliary tasks, namely, ASR error detection and ASR error correction, and we proposed a novel multimodal fusion (MF) method for learning modality-specific and modality-invariant representations across different modalities. Building on this foundation, in this paper, we introduce two additional training strategies. First, we propose an adversarial network to enhance the diversity of modality-specific representations. Second, we introduce a label-based contrastive learning strategy to better capture emotional features. We refer to our proposed method as M4SER and validate its superiority over state-of-the-art methods through extensive experiments using IEMOCAP and MELD datasets.

6 June 2025 in IEEE Transactions on Audio, Speech and Language Processing
Verified
Journal Article
Open access
JHJiajun He
Jiajun He
TTTomoki Toda
Tomoki Toda
End-to-end automatic speech recognition (ASR) models often struggle to accurately recognize rare words. Previously, we introduced an ASR postprocessing method called error detection and context-aware error correction (ED-CEC), which leverages contextual information such as named entities and technical terms to improve the accuracy of ASR transcripts. Although ED-CEC achieves a notable success in correcting rare words, its accuracy remains low when dealing with rare words that have similar pronunciations but different spellings. To address this issue, we proposed a phoneme-augmented multimodal fusion method for context-aware error correction (PMF-CEC) method on the basis of ED-CEC, which allowed for better differentiation between target rare words and homophones. Additionally, we observed that the previous ASR error detection module suffers from overdetection. To mitigate this, we introduced a retention probability mechanism to filter out editing operations with confidence scores below a set threshold, preserving the original operation to improve error detection accuracy. Experiments conducted on five datasets demonstrated that our proposed PMF-CEC maintains reasonable inference speed while further reducing the biased word error rate compared with ED-CEC, showing a stronger advantage in correcting homophones. Moreover, our method outperforms other contextual biasing methods, and remains valuable compared with LLM-based methods in terms of faster inference and better robustness under large biasing lists.

14 January 2026 in IEEE Transactions on Audio, Speech and Language Processing
Verified
Journal Article
Closed access
XSXiaohan Shi
Xiaohan Shi
JHJiajun He
Jiajun He
XLXingfeng Li
Xingfeng Li
TTTomoki Toda
Tomoki Toda
In this paper, we propose an efficient noise-robust approach to noisy speech emotion recognition (NSER). Although conventional NSER methods effectively handle some types of noise, such as stationary noise, they struggle with more complex noise encountered in realistic acoustic environments owing to its complexity and unpredictability. To address this issue, we introduce a novel NSER method that leverages automatic speech recognition (ASR) models as noise-robust feature extractors, filtering out non-vocal information from noisy speech. Specifically, we extract intermediate layer representations from the ASR model to capture emotional speech features for the NSER task. In this study, we conduct a comprehensive evaluation of ASR representations against traditional high-level statistical function (HSF) features, noise reduction approaches, mainstream NSER models, and fine-tuned self-supervised learning (SSL) methods. We analyze encoder and decoder groups through both single-layer and fusion-based multi-layer representations, further investigating the layer-wise performance and robustness of encoder and decoder modules under different noise types and intensities. Additionally, we explore their correlation with ASR performance, assess cross-modal performance using ASR transcripts, and examine cross-lingual generalization. Our experimental results demonstrate that (1) the proposed method outperforms HSF features, noise reduction approaches, mainstream NSER models, and SSL methods; (2) the adapter-based method further enhances ASR representations; (3) higher encoder and lower decoder layers (e.g., in Whisper) yield the best individual performance, while combining encoder and decoder layers across all depths achieves the strongest results; (4) performance decreases as noise intensity increases, especially with human speech noise; (5) this decline mirrors the degradation observed in ASR performance as noise intensity increases; (6) our approach surpasses transcript-based methods using ASR or ground-truth transcriptions; and (7) it achieves robust cross-lingual performance compared with mainstream SSL representations.

19 May 2026 in IEEE Transactions on Biomedical Engineering
Verified
Journal Article
Closed access
DMDing Ma
Ding Ma
JMJinyi Mi
Jinyi Mi
FLFengji Li
Fengji Li
LVLester Phillip Violeta
Lester Phillip Violeta
JHJiajun He
Jiajun He
TTTomoki Toda
Tomoki Toda
Objective: laryngectomees depend on an electromechanical device to generate electrolaryngeal (EL) speech for verbal communication. Compared with normal speech, EL speech suffers from severe distortion, limited phonetic variation, unnatural prosody, and temporal shifts, degrading naturalness and intelligibility. Although sequence-to-sequence (seq2seq) voice conversion (VC) based EL-speech-to-normal-speech conversion (EL2SP) is promising, substantial mismatches between EL and normal speech inevitably cause cumulative mapping errors that limit performance. To address this, we describe a novel representation learning framework integrating speech and text representations to improve mapping and reconstruction quality within a seq2seq VC model. Methods: our methodology comprises two main stages: 1) representation integration and learning, and 2) reconstruction training. A network capable of incorporating auxiliary text information is first constructed with pretrained modules to learn speech–text-based integrated representations. Then, an autoencoder-style reconstruction strategy finalizes EL2SP model to inherit these representations without increasing model complexity. Additional optimization designs are performed across these stages. We introduce three fusion strategies including middle-, input-, and hybrid-level fusion strategies that progressively enhance learning. Moreover, besides standard seq2seq VC objectives, an additional reconstruction loss on the integrated representation is introduced to refine representation transfer. Results: experiments under different EL2SP datasets consistently demonstrate that our methods, combined with data augmentations, outperform baselines relying solely on speech representations regarding both conversion quality and intelligibility. Furthermore, progressive improvements with system design depth validate the effectiveness of our methods. Significance: the proposed methods provide an extensible and practical methodology for EL speech enhancement and assistive communication technologies.