Despite recent advancements in deep learning technologies, Child Speech Recognition remains a challenging task. Current Automatic Speech Recognition (ASR) models require substantial amounts of annotated data for training, which is scarce. In this work, we explore using the ASR model, wav2vec2, with different pretraining and finetuning configurations for self-supervised learning (SSL) toward improving automatic child speech recognition. The pretrained wav2vec2 models were finetuned using different amounts of child speech training data, adult speech data, and a combination of both, to discover the optimum amount of data required to finetune the model for the task of child ASR. Our trained model achieves the best Word Error Rate (WER) of 7.42 on the MyST child speech dataset, 2.91 on the PFSTAR dataset and 12.77 on the CMU KIDS dataset using cleaned variants of each dataset. Our models outperformed the unmodified wav2vec2 BASE 960 on child speech using as little as 10 hours of child speech data in finetuning. The analysis of different types of training data and their effect on inference is provided by using a combination of custom datasets in pretraining, finetuning and inference. These ‘cleaned’ datasets are provided for use by other researchers to provide comparisons with our results.

Studies have shown that newborns are crying differently depending on their need: hunger, tiredness, discomfort, eructation, pain, and so on. Skilled persons such as pediatricians can distinguish between different types of newborn cries and consequently estimate the baby's need by using the sounds and the gestures produced by the baby. However, this is a real problem for the unskilled parents who would like to answer promptly to their baby needs. Recently, research work has been invested into developing automated methods to detect the baby's need using only a short audio recording of his/her live cry. In this paper we report the initial results we obtained by applying various methods that were successful in speaker and language recognition to the task of baby cry recognition. The results are promising: 70% accuracy on a dataset with 39 babies and 5 types of cry.

Speech synthesis has come a long way as current text-to-speech (TTS) models can now generate natural human-sounding speech. However, most of the TTS research focuses on using adult speech data and there has been very limited work done on child speech synthesis. This study developed and validated a training pipeline for fine-tuning state-of-the-art (SOTA) neural TTS models using child speech datasets. This approach adopts a multi-speaker TTS retuning workflow to provide a transfer-learning pipeline. A publicly available child speech dataset was cleaned to provide a smaller subset of approximately 19 hours, which formed the basis of our fine-tuning experiments. Both subjective and objective evaluations were performed using a pretrained MOSNet for objective evaluation and a novel subjective framework for mean opinion score (MOS) evaluations. Subjective evaluations achieved the MOS of 3.95 for speech intelligibility, 3.89 for voice naturalness, and 3.96 for voice consistency. Objective evaluation using a pretrained MOSNet showed a strong correlation between real and synthetic child voices. Speaker similarity was also verified by calculating the cosine similarity between the embeddings of utterances. An automatic speech recognition (ASR) model is also used to provide a word error rate (WER) comparison between the real and synthetic child voices. The final trained TTS model was able to synthesize child-like speech from reference audio samples as short as 5 seconds.

A natural evolution of applications that analyze speech is to improve their robustness to multi-speaker environments. Humans use selective auditory attention and can easily switch focus from one source to another even when listening to a single channel recording with overlapped speech. The same brain feature allows us to detect the number of simultaneously active sources. In order to quantify human level performance for this task we have designed a perception study that evaluates participants' ability to accurately count multiple speakers in a single channel audio file. We also analyzed the influence of listening time and of hearing familiar voices. The study was carried in 3 sessions, with the help of 31, 38 and 80 volunteers and significantly extends the findings in existing literature. Using the conclusions from the perception analysis, several convolutional neural networks were trained to estimate the number of competing speakers on speech timeframes ranging from 25 milliseconds to 1 second. The same models were instructed to tag overlapped speech and we observed F-Score values up to 0.91. For both tasks, the proposed methods lead to lower error than existing approaches and require smaller timeframes. Compared with human listeners, the neural networks can count speakers more accurately by analyzing a considerably shorter recording.