Speaker diarization systems aim to find ‘who spoke when?’ in multi-speaker recordings. The dataset usually consists of meetings, TV/talk shows, telephone and multi-party interaction recordings. In this paper, we propose a novel multimodal speaker diarization technique, which finds the active speaker through audio-visual synchronization model for diarization. A pre-trained audio-visual synchronization model is used to find the synchronization between a visible person and the respective audio. For that purpose, short video segments comprised of face-only regions are acquired using a face detection technique and are then fed to the pre-trained model. This model is a two streamed network which matches audio frames with their respective visual input segments. On the basis of high confidence video segments inferred by the model, the respective audio frames are used to train Gaussian mixture model (GMM)-based clusters. This method helps in generating speaker specific clusters with high probability. We tested our approach on a popular subset of AMI meeting corpus consisting of 5.4 h of recordings for audio and 5.8 h of different set of multimodal recordings. A significant improvement is noticed with the proposed method in term of DER when compared to conventional and fully supervised audio based speaker diarization. The results of the proposed technique are very close to the complex state-of-the art multimodal diarization which shows significance of such simple yet effective technique.

To promote online businesses and sales, e-commerce industry focuses to fulfill users’ demands by giving them top set of recommendations which are ranked through different ranking measures. Deep learning based auto-encoder models have further improved the performance of recommender systems. A state-of-the-art collaborative denoising auto-encoder (CDAE) models user-item interactions as a corrupted version of users rating inputs. However, this architecture still lacks users’ ratings-trend information which is an important parameter to recommend top-N items to users. In this paper, building upon CDAE characteristics, we propose a novel users rating-trend based collaborative denoising auto-encoder (UT-CDAE) which determines user-item correlations by evaluating rating-trend (High or Low) of a user towards a set of items. This inclusion of a user’s rating-trend provides additional regularization flexibility which helps to predict improved top-N recommendations. The correctness of the suggested method is verified through different ranking evaluation metrics i.e., (mean reciprocal rank, mean average precision and normalized discounted gain), for various input corruption values, learning rates and regularization parameters. Experiments on standard ML-100K and ML-1M datasets show that suggested model has improved performance over state-of-the-art denoising auto-encoder models.

Speaker diarization system identifies the speaker homogenous regions in those set of recordings where multiple speakers are present. It answers the question `who spoke when?'. The data set for speaker diarization usually consists of telephone, meetings, TV/ talk shows, broadcast news and other multi-speaker recordings. In this paper, we present the performance of our proposed multimodal speaker diarization system under noisy conditions. Two types of noises comprising additive white Gaussian noise (AWGN) and realistic environmental noise is used to evaluate the system. To mitigate the effect of noise, we propose to add an LSTM based speech enhancement block in our diarization pipeline. This block is trained on synthesized data set with more than 100 noise types to enhance the noisy speech. The enhanced speech is further used in multimodal speaker diarization system which utilizes a pre-trained audio-visual synchronization model to find the active speaker. High confidence active speaker segments are then used to train the speaker specific clusters on the enhanced speech. A subset of AMI corpus consisting of 5.4 h of recordings is used in this analysis. For AWGN, the LSTM model performance improvement is comparable with Wiener filter while in case of realistic environmental noise, the LSTM model improves significantly as compared to Wiener filter in terms of diarization error rate (DER).