This paper presents a new objective quality metric that was adapted to the complex characteristics of immersive video (IV) which is prone to errors caused by processing and compression of multiple input views and virtual view synthesis. The proposed metric, IV-PSNR, contains two techniques that allow for the evaluation of quality loss for typical immersive video distortions: corresponding pixel shift and global component difference. The performed experiments compared the proposal with 31 state-of-the-art quality metrics, showing their performance in the assessment of quality in immersive video coding and processing, and in other applications, using commonly used image quality assessment databases – TID2013 and CVIQ. As presented, IV-PSNR outperforms other metrics in immersive video applications and still can be efficiently used in the evaluation of different images and videos. Moreover, basing the metric on the calculation of PSNR allowed the computational complexity to remain low. Publicly available, efficient implementation of IV-PSNR software was provided by the authors of this paper and is used by ISO/IEC MPEG for evaluation and research on the upcoming MPEG Immersive video (MIV) coding standard.

Free-viewpoint television (FTV) and virtual navigation appear to be hot research topics. In this paper, the authors study the practical development of free-viewpoint television systems that provide the functionality of virtual horizontal navigation around real scenes. The considerations are focused on practical systems that use purely optical depth estimation and might be employed in the next few years. The architectures of such systems are discussed in detail, including acquisition, preprocessing, depth estimation, compression, and presentation. In particular, the optimization of camera locations is discussed, and it is shown that video acquisition using camera pairs is advantageous for scenes with a substantial amount of occlusions. The theoretical considerations are supported by experimental results obtained for standard test multiview video sequences. Furthermore, the paper describes FTV video acquisition systems that consist of modules with pairs of cameras. The modules are sparsely located in arbitrary positions around a scene. Each camera module is equivalent to a video camera with a depth sensor. The hardware requirements, video processing algorithms, and experimental results are reported. In particular, for such systems, a compression technique is discussed that is more efficient than the new three-dimensional HEVC technology. The paper also describes new test video sequences that are obtained from the camera pairs sparsely distributed around scenes.

Delays in the delivery of immersive video to Head Mounted Devices (HMDs) are considered in the paper. The goal of this paper is to analyze the impact of video streaming latency on user experience with HMDs. The paper reports the results of the subjective quality assessment as a function of delay. The possible practical solutions are identified and, on this base, an experimental model of the considered systems is proposed. In order to properly test the influence of the streaming delay on the quality, the extensive subjective tests have been performed. Interesting conclusions have been drawn, showing that human acceptance of delay of translation of virtual point of viewing is much stronger than for delay of rotation of virtual viewing direction in immersive video. Finally, the influence of such observations on the system architectures is concluded.

The selection of optimal views for transmission is critical for the coding efficiency of the MPEG Immersive Video (MIV) profile of Decoder-Side Depth Estimation (DSDE). Standard approaches, which favor a uniform camera distribution, often fail in scenes with complex geometry, leading to decreased quality of depth estimation, and thus, reduced quality of virtual views presented to a viewer. This paper proposes an adaptive view selection method that analyzes the scene’s percentage of occluded regions. Based on this analysis, the encoder dynamically selects a transmission strategy: for scenes with a low occlusion ratio (smaller than 10%), a uniform layout is maintained to maximize spatial coverage; for scenes with a high occlusion ratio, the system switches to grouping cameras into stereo pairs, which are more robust for decreasing numbers of occlusions. Experiments conducted using the TMIV reference software demonstrated that this approach yields measurable quality gains (up to 2 dB BD-IVPSNR) for complex test sequences, such as MartialArts and Frog, without requiring any modifications to the decoder.

Modern video compression is implemented in complex software systems that reuse software modules from various sources. This is particularly evident in experimental software systems designed for researching and standardizing new compression technologies. These systems often incorporate software modules operating in different color spaces. For example, AI-based techniques are often used in video coding experiments. The corresponding software modules often operate on RGB representations, while other modules operate on YC_BC_R components. In this study, we demonstrate that the quality loss resulting from color transformations is comparable to the respective quantization noise. Consecutive cycles of color transformations do not result in significant additional degradation. However, for image compression, very different results are obtained in different color representations. This aspect must be carefully considered in compression research. This paper supports these considerations with extensive experimental results in the context of ITU Recommendations BT.709 and BT.2020, as well as AVC and HEVC compression.