Erik Ekstedt

Erik Ekstedt

Auto-generated profile, based on publicly available information

Erik Ekstedt

Division of Speech, Music and Hearing KTH Royal Institute of Technology Sweden

Research Focus & Metrics

Academic Career Length

7Years (2020-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 January 2022
Verified
Conference Paper
Open access
PDF file
EEErik Ekstedt
Erik Ekstedt
GSGabriel Skantze
Gabriel Skantze
Erik Ekstedt, Gabriel Skantze. Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue. 2022.

10 January 2024 in arXiv
Verified
Preprint
Open access
PDF file
KIKoji Inoue
Koji Inoue
BJBing'Er Jiang
Bing'Er Jiang
EEErik Ekstedt
Erik Ekstedt
TKTatsuya Kawahara
Tatsuya Kawahara
GSGabriel Skantze
Gabriel Skantze
A demonstration of a real-time and continuous turn-taking prediction system is presented. The system is based on a voice activity projection (VAP) model, which directly maps dialogue stereo audio to future voice activities. The VAP model includes contrastive predictive coding (CPC) and self-attention transformers, followed by a cross-attention transformer. We examine the effect of the input context audio length and demonstrate that the proposed system can operate in real-time with CPU settings, with minimal performance degradation.

1 January 2021
Verified
Conference Paper
Open access
PDF file
EEErik Ekstedt
Erik Ekstedt
GSGabriel Skantze
Gabriel Skantze
Erik Ekstedt, Gabriel Skantze. Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue. 2021.

29 May 2023 in arXiv
Verified
Preprint
Open access
PDF file
EEErik Ekstedt
Erik Ekstedt
SWSiyang Wang
Siyang Wang
ÉSÉva Székely
Éva Székely
JGJoakim Gustafson
Joakim Gustafson
GSGabriel Skantze
Gabriel Skantze
Turn-taking is a fundamental aspect of human communication where speakers convey their intention to either hold, or yield, their turn through prosodic cues. Using the recently proposed Voice Activity Projection model, we propose an automatic evaluation approach to measure these aspects for conversational speech synthesis. We investigate the ability of three commercial, and two open-source, Text-To-Speech (TTS) systems ability to generate turn-taking cues over simulated turns. By varying the stimuli, or controlling the prosody, we analyze the models performances. We show that while commercial TTS largely provide appropriate cues, they often produce ambiguous signals, and that further improvements are possible. TTS, trained on read or spontaneous speech, produce strong turn-hold but weak turn-yield cues. We argue that this approach, that focus on functional aspects of interaction, provides a useful addition to other important speech metrics, such as intelligibility and naturalness.

21 October 2020 in arXiv
Verified
Preprint
Open access
PDF file
EEErik Ekstedt
Erik Ekstedt
GSGabriel Skantze
Gabriel Skantze
Syntactic and pragmatic completeness is known to be important for turn-taking prediction, but so far machine learning models of turn-taking have used such linguistic information in a limited way. In this paper, we introduce TurnGPT, a transformer-based language model for predicting turn-shifts in spoken dialog. The model has been trained and evaluated on a variety of written and spoken dialog datasets. We show that the model outperforms two baselines used in prior work. We also report on an ablation study, as well as attention and gradient analyses, which show that the model is able to utilize the dialog context and pragmatic completeness for turn-taking prediction. Finally, we explore the model's potential in not only detecting, but also projecting, turn-completions.

15 March 2024 in arXiv
Verified
Preprint
Open access
PDF file
KIKoji Inoue
Koji Inoue
BJBing'Er Jiang
Bing'Er Jiang
EEErik Ekstedt
Erik Ekstedt
TKTatsuya Kawahara
Tatsuya Kawahara
GSGabriel Skantze
Gabriel Skantze
This paper investigates the application of voice activity projection (VAP), a predictive turn-taking model for spoken dialogue, on multilingual data, encompassing English, Mandarin, and Japanese. The VAP model continuously predicts the upcoming voice activities of participants in dyadic dialogue, leveraging a cross-attention Transformer to capture the dynamic interplay between participants. The results show that a monolingual VAP model trained on one language does not make good predictions when applied to other languages. However, a multilingual model, trained on all three languages, demonstrates predictive performance on par with monolingual models across all languages. Further analyses show that the multilingual model has learned to discern the language of the input signal. We also analyze the sensitivity to pitch, a prosodic cue that is thought to be important for turn-taking. Finally, we compare two different audio encoders, contrastive predictive coding (CPC) pre-trained on English, with a recent model based on multilingual wav2vec 2.0 (MMS).