Odette Scharenborg

Odette Scharenborg

Auto-generated profile, based on publicly available information

Odette Scharenborg

Delft University of Technology The Netherlands

Research Focus & Metrics

Academic Career Length

9Years (2018-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
20 September 2023 in Computer Speech & Language
Verified
Journal Article
Open access
SFSiyuan Feng
Siyuan Feng
BHBence Mark Halpern
Bence Mark Halpern
OKOlya Kudina
Olya Kudina
OSOdette Scharenborg
Odette Scharenborg

12 October 2022 in IEEE Transactions on Multimedia
Verified
Journal Article
Closed access
XWXinsheng Wang
Xinsheng Wang
QXQicong Xie
Qicong Xie
Jihua ZhuJihua Zhu
Jihua Zhu
LXLei Xie
Lei Xie
OSOdette Scharenborg
Odette Scharenborg
Automatically generating videos in which synthesized speech is synchronized with lip movements in a talking head has great potential in many human-computer interaction scenarios. In this paper, we present an automatic method to generate synchronized speech and talking-head videos on the basis of text and a single face image of an arbitrary person as input. In contrast to previous text-driven talking head generation methods, which can only synthesize the voice of a specific person, the proposed method is capable of synthesizing speech for any person. Specifically, the proposed method decomposes the generation of synchronized speech and talking head videos into two stages, i.e., a text-to-speech (TTS) stage and a speech-driven talking head generation stage. The proposed TTS module is a face-conditioned multi-speaker TTS model that gets the speaker identity information from face images instead of speech, which allows us to synthesize a personalized voice on the basis of the input face image. To generate the talking head videos from the face images, a facial landmark-based method that can predict both lip movements and head rotations is proposed. Extensive experiments demonstrate that the proposed method is able to generate synchronized speech and talking head videos for arbitrary persons, in which the timbre of the synthesized voice is in harmony with the input face, and the proposed landmark-based talking head method outperforms the state-of-the-art landmark-based method on generating natural talking head videos.

13 February 2020 in IEEE/ACM Transactions on Audio, Speech, and Language Processing
Verified
Journal Article
Closed access
OSOdette Scharenborg
Odette Scharenborg
LBLaurent Besacier
Laurent Besacier
ABAlan W. Black
Alan W. Black
MHMark Hasegawa-Johnson
Mark Hasegawa-Johnson
FMFlorian Metze
Florian Metze
EDEmmanuel Dupoux
Emmanuel Dupoux
Speech technology plays an important role in our everyday life. Among others, speech is used for human-computer interaction, for instance for information retrieval and on-line shopping. In the case of an unwritten language, however, speech technology is unfortunately difficult to create, because it cannot be created by the standard combination of pre-trained speech-to-text and text-to-speech subsystems. The research presented in this paper takes the first steps towards speech technology for unwritten languages. Specifically, the aim of this work was 1) to learn speech-to-meaning representations without using text as an intermediate representation, and 2) to test the sufficiency of the learned representations to regenerate speech or translated text, or to retrieve images that depict the meaning of an utterance in an unwritten language. The results suggest that building systems that go directly from speech-to-meaning and from meaning-to-speech, bypassing the need for text, is possible.

22 January 2021 in IEEE/ACM Transactions on Audio, Speech, and Language Processing
Verified
Journal Article
Closed access
XWXinsheng Wang
Xinsheng Wang
TQTingting Qiao
Tingting Qiao
Jihua ZhuJihua Zhu
Jihua Zhu
AHAlan Hanjalic
Alan Hanjalic
OSOdette Scharenborg
Odette Scharenborg
Text-based technologies, such as text translation from one language to another, and image captioning, are gaining popularity. However, approximately half of the world's languages are estimated to be lacking a commonly used written form. Consequently, these languages cannot benefit from text-based technologies. This paper presents 1) a new speech technology task, i.e., a speech-to-image generation (S2IG) framework which translates speech descriptions to photo-realistic images 2) without using any text information, thus allowing unwritten languages to potentially benefit from this technology. The proposed speech-to-image framework, referred to as S2IGAN, consists of a speech embedding network and a relation-supervised densely-stacked generative model. The speech embedding network learns speech embeddings with the supervision of corresponding visual information from images. The relation-supervised densely-stacked generative model synthesizes images, conditioned on the speech embeddings produced by the speech embedding network, that are semantically consistent with the corresponding spoken descriptions. Extensive experiments are conducted on four public benchmark databases: two databases that are commonly used in text-to-image generation tasks, i.e., CUB-200 and Oxford-102 for which we created synthesized speech descriptions, and two databases with natural speech descriptions which are often used in the field of cross-modal learning of speech and images, i.e., Flickr8k and Places. Results on these databases demonstrate the effectiveness of the proposed S2IGAN on synthesizing high-quality and semantically-consistent images from the speech signal, yielding a good performance and a solid baseline for the S2IG task.

9 February 2022 in Computer Speech & Language
Verified
Journal Article
Closed access
PŻPiotr Żelasko
Piotr Żelasko
SFSiyuan Feng
Siyuan Feng
LMLaureano Moro-Velazquez
Laureano Moro-Velazquez
AAAli Abavisani
Ali Abavisani
SBSaurabhchand Bhati
Saurabhchand Bhati
NDNajim Dehak
Najim Dehak

16 March 2022 in Cortex
Verified
Journal Article
Closed access
ASAntje Strauß
Antje Strauß
TWTongyu Wu
Tongyu Wu
JMJames M. McQueen
James M. McQueen
OSOdette Scharenborg
Odette Scharenborg
FHFlorian Hintz
Florian Hintz