Sefik Emre Eskimez

Sefik Emre Eskimez

Auto-generated profile, based on publicly available information

Sefik Emre Eskimez

Microsoft Speech and Dialogue Research Group, USA

Research Focus & Metrics

Academic Career Length

11Years (2016-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
26 July 2021 in IEEE Transactions on Multimedia
Verified
Journal Article
Closed access
SESefik Emre Eskimez
Sefik Emre Eskimez
YZYou Zhang
You Zhang
ZDZhiyao Duan
Zhiyao Duan
Visual emotion expression plays an important role in audiovisual speech communication. In this work, we propose a novel approach to rendering visual emotion expression in speech-driven talking face generation. Specifically, we design an end-to-end talking face generation system that takes a speech utterance, a single face image, and a categorical emotion label as input to render a talking face video synchronized with the speech and expressing the conditioned emotion. Objective evaluation on image quality, audiovisual synchronization, and visual emotion expression shows that the proposed system outperforms a state-of-the-art baseline system. Subjective evaluation of visual emotion expression and video realness also demonstrates the superiority of the proposed system. Furthermore, we conduct a human emotion recognition pilot study using generated videos with mismatched emotions among the audio and visual modalities. Results show that humans respond to the visual modality more significantly than the audio modality on this task.

9 May 2023 in ACM Transactions on Asian and Low-Resource Language Information Processing
Verified
Journal Article
Open access
JLJunwei Liao
Junwei Liao
SESefik Emre Eskimez
Sefik Emre Eskimez
LLLiyang Lu
Liyang Lu
YSYu Shi
Yu Shi
MGMing Gong
Ming Gong
MZMichael Zeng
Michael Zeng
Modern Automatic Speech Recognition (ASR) systems can achieve high performance in terms of recognition accuracy. However, a perfectly accurate transcript still can be challenging to read due to grammatical errors, disfluency, and other noises common in spoken communication. These readable issues introduced by speakers and ASR systems will impair the performance of downstream tasks and the understanding of human readers. In this work, we present a task called ASR post-processing for readability (APR) and formulate it as a sequence-to-sequence text generation problem. The APR task aims to transform the noisy ASR output into a readable text for humans and downstream tasks while maintaining the semantic meaning of speakers. We further study the APR task from the benchmark dataset, evaluation metrics, and baseline models: First, to address the lack of task-specific data, we propose a method to construct a dataset for the APR task by using the data collected for grammatical error correction. Second, we utilize metrics adapted or borrowed from similar tasks to evaluate model performance on the APR task. Lastly, we use several typical or adapted pre-trained models as the baseline models for the APR task. Furthermore, we fine-tune the baseline models on the constructed dataset and compare their performance with a traditional pipeline method in terms of proposed evaluation metrics. Experimental results show that all the fine-tuned baseline models perform better than the traditional pipeline method, and our adapted RoBERTa model outperforms the pipeline method by 4.95 and 6.63 BLEU points on two test sets, respectively. The human evaluation and case study further reveal the ability of the proposed model to improve the readability of ASR transcripts.

6 June 2018
Verified
Book Chapter
Closed access
PDF file
SESefik Emre Eskimez
Sefik Emre Eskimez
RMRoss K. Maddox
Ross K. Maddox
CXChenliang Xu
Chenliang Xu
ZDZhiyao Duan
Zhiyao Duan

1 May 2018 in Speech Communication
Verified
Journal Article
Closed access
SESefik Emre Eskimez
Sefik Emre Eskimez
PSPeter Soufleris
Peter Soufleris
ZDZhiyao Duan
Zhiyao Duan
WHWendi Heinzelman
Wendi Heinzelman

16 October 2019 in IEEE/ACM Transactions on Audio, Speech, and Language Processing
Verified
Journal Article
Closed access
SESefik Emre Eskimez
Sefik Emre Eskimez
RMRoss K. Maddox
Ross K. Maddox
CXChenliang Xu
Chenliang Xu
ZDZhiyao Duan
Zhiyao Duan
Visual cues such as lip movements, when available, play an important role in speech communication. They are especially helpful for the hearing impaired population or in noisy environments. When not available, having a system to automatically generate talking faces in sync with input speech would enhance speech communication and enable many novel applications. In this article, we present a new system that can generate 3D talking face landmarks from speech in an online fashion. We employ a neural network that accepts the raw waveform as an input. The network contains convolutional layers with 1D kernels and outputs the active shape model (ASM) coefficients of face landmarks. To promote smoother transitions between video frames, we present a variant of the model that has the same architecture but also accepts the previous frame's ASM coefficients as an additional input. To cope with background noise, we propose a new training method to incorporate speech enhancement ideas at the feature level. Objective evaluations on landmark prediction show that the proposed system yields statistically significantly smaller errors than two state-of-the-art baseline methods on both a single-speaker dataset and a multi-speaker dataset. Experiments on noisy speech input with five types of non-stationary unseen noise show statistically significant improvements of the system performance thanks to the noise-resilient training method. Finally, subjective evaluations show that the generated talking faces have a significantly more convincing match with the input audio, achieving a similarly convincing level of realism as the ground-truth landmarks.

1 March 2016
Verified
Conference Paper
Open access
SESefik Emre Eskimez
Sefik Emre Eskimez
KIKenneth Imade
Kenneth Imade
NYNa Yang
Na Yang
MSMelissa Sturge-Apple
Melissa Sturge-Apple
ZDZhiyao Duan
Zhiyao Duan
WHWendi Heinzelman
Wendi Heinzelman
The fact that emotions play a vital role in social interactions, along with the demand for novel human-computer interaction applications, have led to the development of a number of automatic emotion classification systems. However, it is still debatable whether the performance of such systems can compare with human coders. To address this issue, in this study, we present a comprehensive comparison in a speech-based emotion classification task between 138 Amazon Mechanical Turk workers (Turkers) and a state-of-the-art automatic computer system. The comparison includes classifying speech utterances into six emotions (happy, neutral, sad, anger, disgust and fear), into three arousal classes (active, passive, and neutral), and into three valence classes (positive, negative, and neutral). The results show that the computer system outperforms the naive Turkers in almost all cases. Furthermore, the computer system can increase the classification accuracy by rejecting to classify utterances for which it is not confident, while the Turkers do not show a significantly higher classification accuracy on their confident utterances versus unconfi-dent ones.