Deep learning can be used for audio signal classification in a variety of ways. It can be used to detect and classify various types of audio signals such as speech, music, and environmental sounds. Deep learning models are able to learn complex patterns of audio signals and can be trained on large datasets to achieve high accuracy. To employ deep learning for audio signal classification, the audio signal must first be represented in a suitable form. This can be done using signal representation techniques such as using spectrograms, Mel-frequency Cepstral coefficients, linear predictive coding, and wavelet decomposition. Once the audio signal is represented in a suitable form, it can then be fed into a deep learning model. Various deep learning models can be utilized for audio classification. We provide an extensive survey of current deep learning models used for a variety of audio classification tasks. In particular, we focus on works published under five different deep neural network architectures, namely Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Autoencoders, Transformers and Hybrid Models (hybrid deep learning models and hybrid deep learning models with traditional classifiers). CNNs can be used to classify audio signals into different categories such as speech, music, and environmental sounds. They can also be used for speech recognition, speaker identification, and emotion recognition. RNNs are widely used for audio classification and audio segmentation. RNN models can capture temporal patterns of audio signals and be used to classify audio segments into different categories. Another approach is to use autoencoders for learning the features of audio signals and then classifying the signals into different categories. Transformers are also well-suited for audio classification. In particular, temporal and frequency features can be extracted to identify the characteristics of the audio signals. Finally, hybrid models for audio classification either combine various deep learning architectures (i.e. CNN-RNN) or combine deep learning models with traditional machine learning techniques (i.e. CNN-Support Vector Machine). These hybrid models take advantage of the strengths of different architectures while avoiding their weaknesses. Existing literature under different categories of deep learning are summarized and compared in detail.

Emotion information from speech can effectively help robots understand speaker’s intentions in natural human-robot interaction. The human auditory system can easily track temporal dynamics of emotion by perceiving the intensity and fundamental frequency of speech, and focus on the salient emotion regions. Therefore, speech emotion recognition combined with the auditory mechanism and attention mechanism may be an effective way. Some previous studies used auditory-based static features to identify emotion while ignoring the emotion dynamics. Some other studies used attention models to capture the salient regions of emotion while ignoring cognitive continuity. To fully utilize the auditory and attention mechanism, we first investigate temporal modulation cues from auditory front-ends and then propose a joint deep learning model that combines 3D convolutions and attention-based sliding recurrent neural networks (ASRNNs) for emotion recognition. Our experiments on the IEMOCAP and MSP-IMPROV datasets indicate that the proposed method can be effectively used to recognize the emotions of speech from temporal modulation cues. The subjective evaluation shows that the attention patterns of the attention model are basically consistent with human behaviors in recognizing the emotions.

Although the rounded-exponential (roex) filter has been successfully used to represent the magnitude response of the auditory filter, recent studies with the $\mathrm{roex}(p,w,t)$ filter reveal two serious problems: the fits to notched-noise masking data are somewhat unstable unless the filter is reduced to a physically unrealizable form, and there is no time-domain version of the $\mathrm{roex}(p,w,t)$ filter to support modeling of the perception of complex sounds. This paper describes a compressive gammachirp (cGC) filter with the same architecture as the $\mathrm{roex}(p,w,t)$ which can be implemented in the time domain. The gain and asymmetry of this parallel cGC filter are shown to be comparable to those of the $\mathrm{roex}(p,w,t)$ filter, but the fits to masking data are still somewhat unstable. The $\mathrm{roex}(p,w,t)$ and parallel cGC filters were also compared with the cascade cGC filter [Patterson et al., J. Acoust. Soc. Am. 114, 1529–1542 (2003)], which was found to provide an equivalent fit with 25% fewer coefficients. Moreover, the fits were stable. The advantage of the cascade cGC filter appears to derive from its parsimonious representation of the high-frequency side of the filter. It is concluded that cGC filters offer better prospects than roex filters for the representation of the auditory filter.

The rise of synthetic speech technologies has triggered growing concerns about the increasing difficulty in distinguishing between real and fake voices. In this context, we propose novel hybrid transformer-based models together with different audio feature analysis techniques and achieved the state-of-the-art results. To the best of our knowledge, none of the existing methods have considered combining various hybrid transformer models together with different audio features for fake speech classification, which forms the main novelty of our work. In our work, transformer models are compared with hybrid transformer architectures including Convolutional Neural Network (CNN)-Transformer (i.e., ResNet34- Transformer and VGG16-Transformer models), Bi-directional Long Short-Term Memory (Bi-LSTM)- Transformer, and Transformer with Support Vector Machine (SVM) using different audio feature extraction techniques. In our approach, we utilize three audio attribute extraction techniques: Mel spectrogram (Mel), Mel Frequency Cepstral Coefficient (MFCC), and Short-Time Fourier Transform (STFT) as input representations. The results of our evaluation with instances of real and fake speech using the ASVspoof LA dataset with hybrid transformer models across various audio features indicate that the STFT feature performs best with the ResNet34-Transformer model, achieving a state-of-the-art performance with a development set equal error rate (EER) of 0.0% and evaluation set EER of 3.22%, surpassing all other methods. In terms of accuracy, the STFT feature also performed best with the VGG16-Transformer model, achieving a development set accuracy of 99.55% and an evaluation set accuracy of 94.04%. These results indicate that the proposed study achieved better performance compared to the baseline and state-of-the-art methods.

Currently, speech emotion recognition models still could not show satisfactory performance due to the complexity of emotions. In most of the previous studies, there is a common problem that some of the particular emotions are severely misclassified. In this paper, we propose a novel framework integrating cascaded attention network and adversarial joint loss strategy for speech emotion recognition, aiming at discriminating the confusions by emphasizing more on the emotions which are difficult to be correctly classified. First, we extract log-Mels, deltas and delta-deltas of log-Mels as 3D features to effectively reduce the interference of external factors. Next, we introduce a cascaded attention network to extract effective emotional features, where spatiotemporal attention selectively locates the targeted emotional regions from the input features. In these targeted regions, the self attention with head fusion captures the long-distance dependence of temporal features. Finally, an adversarial joint loss strategy is proposed to distinguish the emotional embeddings with high similarity by the generated hard triplets in an adversarial fashion. To evaluate our proposed method, experiments are performed with the IEMOCAP, CASIA, and EMODB corpora. The experimental results demonstrate that our proposed method significantly outperforms the state-of-the-art approaches on all datasets.