360-degree video provides an immersive experience to end-users through Virtual Reality (VR) Head-Mounted-Displays (HMDs). However, it is not trivial to understand the Quality of Experience (QoE) of 360-degree video since user experience is influenced by various factors that affect QoE when watching a 360-degree video in VR. This manuscript presents a machine learning-based QoE prediction of 360-degree video in VR, considering the two key QoE aspects: perceptual quality and cybersickness. In addition, we proposed two new QoE-affecting factors: user’s familiarity with VR and user’s interest in 360-degree video for the QoE evaluation. To aim this, we first conduct a subjective experiment on 96 video samples and collect datasets from 29 users for perceptual quality and cybersickness. We design a new Logistic Regression (LR) based model for QoE prediction in terms of perceptual quality. The prediction accuracy of the proposed model is compared against well-known supervised machine-learning algorithms such as k-Nearest Neighbors (kNN), Support Vector Machine (SVM), and Decision Tree (DT) with respect to accuracy rate, recall, f1-score, precision, and mean absolute error (MAE). LR performs well with 86% accuracy, which is in close agreement with subjective opinion. The prediction accuracy of the proposed model is then compared with existing QoE models in terms of perceptual quality. Finally, we build a Neural Network-based model for the QoE prediction in terms of cybersickness. The proposed model performs well against the state of the art QoE prediction methods in terms of cybersickness.

360-degree Virtual Reality (VR) videos have already taken up viewers’ attention by storm. Despite the immense attractiveness and hype, VR conveys a loathsome side effect called “cybersickness” that often creates significant discomfort to the viewers. It is of great importance to evaluate the factors that induce cybersickness symptoms and its deterioration on the end user’s Quality-of-Experience (QoE) when visualizing 360-degree videos in VR. This manuscript’s intent is to subjectively investigate factors of high priority that affect a user’s QoE in terms of perceptual quality, presence, and cybersickness. The content type (fast, medium, and slow), the effect of camera motion (fixed, horizontal, and vertical), and the number of moving targets (none, single, and multiple) in a video can be the factors that may affect the QoE. The significant effect of such factors on end-user QoE under various stalling events (none, single, and multiple) is evaluated in a subjective experiment. The results from subjective experiments show a notable impact of these factors on end-user QoE. Finally, to label the viewing safety concern in VR, we propose a neural network-based QoE prediction method that can predict the degree of cybersickness influenced by 360-degree videos under various stalling events in VR. The performance accuracy of the proposed method is then compared against well-known Machine Learning (ML) algorithms and existing QoE prediction models. The proposed method achieved a 90% prediction accuracy rate and performed well against existing models and other ML methods.

Automatic facial expression recognition is an emerging field. Moreover, the interest has been increased with the transition from laboratory-controlled conditions to in the wild scenarios. Most of the research has been done over nonoccluded faces under the constrained environment, while automatic facial expression is less understood/implemented for partial occlusion in the real world conditions. Apart from that, our research aims to tackle the issues of overfitting (caused by the shortage of adequate training data) and to alleviate the expression-unrelated/intraclass/nonlinear facial variations, such as head pose estimation, eye gaze estimation, intensity and microexpressions. In our research, we control the magnitude of each Action Unit (AU) and combine several of the Action Unit combinations to leverage learning from the generative and discriminative representations for automatic FER. We have also addressed the problem of diversification of expressions from lab controlled to real-world scenarios from our cross-database study and proposed a model for enhancement of the discriminative power of deep features while increasing the interclass scatters, by preserving the locality closeness. Furthermore, facial expression consists of an expressive component as well as neutral component, so we proposed a generative model which is capable of generating neutral expression from an input image using cGAN. The expressive component is filtered and passed to the intermediate layers and the process is called De-expression Residue Learning. The residue in the intermediate/middle layers is very important for learning through expressive components. Finally, we validate the effectiveness of our method (DLP-DeRL) through qualitative and quantitative experimental results using four databases. Our method is more accurate and robust, and outperforms all the existing methods (hand crafted features and deep learning) while dealing the images in the wild.

The interest in the facial expression recognition (FER) is increasing day by day due to its practical and potential applications, such as human physiological interaction diagnosis and mental diseases detection. This area has received much attention from the research community in recent years and achieved remarkable results; however, a significant improvement is required in spatial problems. This research work presents a novel framework and proposes an effective and robust solution for FER under an unconstrained environment. Face detection is performed using the supervision of facial attributes. Faceness‐Net is used for deep facial part responses for the detection of faces under severe unconstrained variations. In order to improve the generalization problems and avoid insufficient data regime, Deep Convolutional Graphical Adversarial Network (DC‐GAN) is utilized. Due to the challenging environmental factors faced in the wild, a large number of noises disrupt feature extraction, thus making it hard to capture ground truth. We leverage different multimodal sensors with a camera that aids in data acquisition, by extracting the features more accurately and improve the overall performance of FER. These intelligent sensors are used to tackle the significant challenges like illumination variance, subject dependence, and head pose. Dual‐enhanced capsule network is used which is able to handle the spatial problem. The traditional capsule networks are unable to sufficiently extract the features, as the distance varies greatly between facial features. Therefore, the proposed network is capable of spatial transformation due to action unit aware mechanism and thus forward most desiring features for dynamic routing between capsules. Squashing function is used for the classification function. We have elaborated the effectiveness of our method by validating the results on four popular and versatile databases that outperform all state‐of‐the‐art methods.

Self-supervised learning (SSL) has improved the performance in various speech processing downstream tasks. Pretraining speech representations with SSL models requires large datasets which can be hard to collect for low resource language scenarios. To address this problem we propose to use audio augmentation to pre-train the SSL model with a smaller dataset but augmented. We illustrate our technique using autoregressive predictive coding (APC) by using augmentation effects directly at an audio level in order to generate synthetic training data. As a result, we aim to increase robustness in trained SSL model. We apply two kinds of audio augmentation, speed perturbation and speed perturbation with pitch change. We then evaluate our pre-trained APC model on a downstream phoneme recognition task. From our results and analysis, we conclude that applying augmentation in a low resource configurations can improve the robustness in the pre-trained model resulting in improved phoneme classification accuracy for a downstream task.