We recently presented a new model for singing synthesis based on a modified version of the WaveNet architecture. Instead of modeling raw waveform, we model features produced by a parametric vocoder that separates the influence of pitch and timbre. This allows conveniently modifying pitch to match any target melody, facilitates training on more modest dataset sizes, and significantly reduces training and generation times. Nonetheless, compared to modeling waveform directly, ways of effectively handling higher-dimensional outputs, multiple feature streams and regularization become more important with our approach. In this work, we extend our proposed system to include additional components for predicting F0 and phonetic timings from a musical score with lyrics. These expression-related features are learned together with timbrical features from a single set of natural songs. We compare our method to existing statistical parametric, concatenative, and neural network-based approaches using quantitative metrics as well as listening tests.
This paper introduces the concept of synthesis based on performance sampling. It explains that although sampling has been considered a way to capture and reproduce the sound of an instrument, it should be better considered a way to model the sonic space produced by a performer with an instrument. The paper presents a singing voice synthesizer, pointing out the main issues and complexities emerging along its design. Although the current system is able to generate convincing results in certain situations, there is still much room for improvements, especially in the areas of expression, spectral modeling and sonic space design. However, computer singing is definitely coming close to becoming indistinguishable from human performances
14 October 2015 in IEEE Signal Processing Magazine
In the context of singing voice synthesis, expression control manipulates a set of voice features related to a particular emotion, style, or singer. Also known as performance modeling, it has been approached from different perspectives and for different purposes, and different projects have shown a wide extent of applicability. The aim of this article is to provide an overview of approaches to expression control in singing voice synthesis. We introduce some musical applications that use singing voice synthesis techniques to justify the need for an accurate control of expression. Then, expression is defined and related to speech and instrument performance modeling. Next, we present the commonly studied set of voice parameters that can change perceptual aspects of synthesized voices. After that, we provide an up-to-date classification, comparison, and description of a selection of approaches to expression control. Then, we describe how these approaches are currently evaluated and discuss the benefits of building a common evaluation framework and adopting perceptually-motivated objective measures. Finally, we discuss the challenges that we currently foresee.
This article deals with automatic transcription of flamenco music recordings—more specifically, a cappella singing. We first study the specifics of flamenco singing and propose a transcription system based on fundamental frequency and energy estimation, which incorporates an iterative strategy for note segmentation and labeling. The proposed approach is evaluated on a music collection of 72 performances, including a variety of singers and recording conditions, and the presence or absence of percussion, background voices, and noise. We obtain satisfying results for the different approaches tested, and our system outperforms a state-of-the-art approach designed for other singing styles. In this study, we discuss the difficulties found in transcribing flamenco singing and in evaluating the obtained transcriptions, we analyze the influence of the different steps of the algorithm, and we state the main limitations of our approach and discuss challenges for future studies.