Recent advances in media capture and processing technologies have enabled new forms of true 3-D media content that increase the degree of user immersion. The demand for more engaging forms of entertainment means that content distributors and broadcasters need to fine-tune their delivery mechanisms over the Internet as well as develop new models for quantifying and predicting user experience of these new forms of content. In the work described in this paper, we undertake one of the first studies into the quality of experience (QoE) of real-time 3-D media content streamed to virtual reality (VR) headsets for entertainment purposes, in the context of game spectating. Our focus is on tele-immersive media that embed real users within virtual environments of interactive games. A key feature of engaging and realistic experiences in full 3-D media environments, is allowing users unrestricted viewpoints. However, this comes at the cost of increased network bandwidth and the need of limiting network effects in order to transmit a realistic, real-time representation of the participants. The visual quality of 3-D media is affected by geometry and texture parameters while the temporal aspects of smooth movement and synchronization are affected by lag introduced by network transmission effects. In this paper, we investigate varying network conditions for a set of tele-immersive media sessions produced in a range of visual quality levels. Further, we investigate user navigation issues that inhibit free viewpoint VR spectating of live 3-D media. After reporting on a study with multiple users we analyze the results and assess the overall QoE with respect to a range of visual quality and latency parameters. We propose a neural network QoE prediction model for 3-D media, constructed from a combination of visual and network parameters.

This work provides a systematic understanding of the requirements of live 3D mesh coding, targeting (tele-)immersive media streaming applications. We thoroughly benchmark in rate-distortion and runtime performance terms, four static 3D mesh coding solutions that are openly available. Apart from mesh geometry and connectivity, our analysis includes experiments for compressing vertex normals and attributes, something scarcely found in literature. Additionally, we provide a theoretical model of the tele-immersion pipeline that calculates its expected frame-rate, as well as lower and upper bounds for its end-to-end latency. In order to obtain these measures, the theoretical model takes into account the compression performance of the codecs and some indicative network conditions. Based on the results we obtained through our codec benchmarking, we used our theoretical model to calculate and provide concrete measures for these tele-immersion pipeline’s metrics and discuss on the optimal codec choice depending on the network setup. This offers deep insight into the available solutions and paves the way for future research.

An important line of research attempts to explain CNN image classifier predictions and intermediate layer representations in terms of human understandable concepts. In this work, we expand on previous works in the literature that use annotated concept datasets to extract interpretable feature space directions and propose an unsupervised post-hoc method to extract a disentangling interpretable basis by looking for the rotation of the feature space that explains sparse one-hot thresholded transformed representations of pixel activations. We do experimentation with existing popular CNNs and demonstrate the effectiveness of our method in extracting an interpretable basis across network architectures and training datasets. We make extensions to the existing basis interpretability metrics found in the literature and show that, intermediate layer representations become more interpretable when transformed to the bases extracted with our method. Finally, using the basis interpretability metrics, we compare the bases extracted with our method with the bases derived with a supervised approach and find that, in one aspect, the proposed unsupervised approach has a strength that constitutes a limitation of the supervised one and give potential directions for future research.