3 August 2022 in IEEE Access
Fueled by advances in multi-party communications, increasingly mature immersive technologies being adopted, and the COVID-19 pandemic, a new wave of social virtual reality (VR) platforms have emerged to support socialization, interaction, and collaboration among multiple remote users who are integrated into shared virtual environments. Social VR aims to increase levels of (co-)presence and interaction quality by overcoming the limitations of 2D windowed representations in traditional multi-party video conferencing tools, although most existing solutions rely on 3D avatars to represent users. This article presents a social VR platform that supports real-time volumetric holographic representations of users that are based on point clouds captured by off-the-shelf RGB-D sensors, and it analyzes the platform’s potential for conducting interactive holomeetings (i.e., holoconferencing scenarios). This work evaluates such a platform’s performance and readiness for conducting meetings with up to four users, and it provides insights into aspects of the user experience when using single-camera and low-cost capture systems in scenarios with both frontal and side viewpoints. Overall, the obtained results confirm the platform’s maturity and the potential of holographic communications for conducting interactive multi-party meetings, even when using low-cost systems and single-camera capture systems in scenarios where users are sitting or have a limited translational movement along the X, Y, and Z axes within the 3D virtual environment (commonly known as 3 Degrees of Freedom plus, 3DoF+).
9 November 2021 in IEEE Transactions on Broadcasting
Social Virtual Reality (VR) allows multiple distributed users getting together in shared virtual environments to socially interact and collaborate. This article explores the applicability and potential of Social VR in the broadcast sector, focusing on a live TV show use case, by providing three main contributions: 1) a novel and lightweight social VR platform; 2) a professional piece of VR content to recreate an interactive live TV show; and 3) an analysis of the performance and user experience. The Social VR platform includes different innovative and outstanding features compared to state-of-the-art solutions. It allows a real-time integration of remote users in shared virtual environments, using realistic volumetric representations and affordable capturing systems, thus not relying on the use of synthetic avatars. It supports a seamless and rich integration of heterogeneous media formats, including 3D scenarios, dynamic volumetric representation of users and (live/stored) stereoscopic 2D and 180°/360° videos. In addition, it enables low-latency interaction between volumetric users and a video-based presenter (Chroma keying) and a dynamic control of the media playout to adapt to the session’s evolution. The article also describes the production process of an immersive an interactive TV show to demonstrate the platform’s capabilities and its potential benefits. On the one hand, the results from objective tests show the satisfactory performance of the platform. On the other hand, the promising results from user tests support the potential impact of the presented platform, opening up new opportunities in the broadcast sector.
9 October 2012 in IEEE Transactions on Circuits and Systems for Video Technology
Within free viewpoint video, the 3-D reconstruction of the scene is created from a high number of viewpoints. Every viewpoint is represented by a traditional sequence, called texture, and its associated depth information. This is known as a View plus Depth environment. In this paper, a novel low complexity mode decision and motion estimation algorithm for the H.264/AVC based encoding of depth sequences is proposed. Given that a texture sequence and its associated depth represent the same scene from the same point of view, they should have similar motion characteristics. The complexity reduction of the depth encoding, in the proposed algorithm, is obtained by taking advantage of the texture motion information that has been previously processed by a traditional H.264/AVC encoder. The characteristics of depth and texture sequences are analyzed, focusing on similarities and differences that are properly managed to design an algorithm able to detect when the motion information of the texture might be usefully exploited in the encoding of the corresponding depth sequence. The proposed method is able to achieve the same objective quality, measured by means of the PSNR and the VQM, than a traditional H.264/AVC encoder with a reduction of up to 58% of the computational burden.