This article introduces the ISO/IEC MPEG Immersive Video (MIV) standard, MPEG-I Part 12, which is undergoing standardization. The draft MIV standard provides support for viewing immersive volumetric content captured by multiple cameras with six degrees of freedom (6DoF) within a viewing space that is determined by the camera arrangement in the capture rig. The bitstream format and decoding processes of the draft specification along with aspects of the Test Model for Immersive Video (TMIV) reference software encoder, decoder, and renderer are described. The use cases, test conditions, quality assessment methods, and experimental results are provided. In the TMIV, multiple texture and geometry views are coded as atlases of patches using a legacy 2-D video codec, while optimizing for bitrate, pixel rate, and quality. The design of the bitstream format and decoder is based on the visual volumetric video-based coding (V3C) and video-based point cloud compression (V-PCC) standard, MPEG-I Part 5.

This paper presents a new objective quality metric that was adapted to the complex characteristics of immersive video (IV) which is prone to errors caused by processing and compression of multiple input views and virtual view synthesis. The proposed metric, IV-PSNR, contains two techniques that allow for the evaluation of quality loss for typical immersive video distortions: corresponding pixel shift and global component difference. The performed experiments compared the proposal with 31 state-of-the-art quality metrics, showing their performance in the assessment of quality in immersive video coding and processing, and in other applications, using commonly used image quality assessment databases – TID2013 and CVIQ. As presented, IV-PSNR outperforms other metrics in immersive video applications and still can be efficiently used in the evaluation of different images and videos. Moreover, basing the metric on the calculation of PSNR allowed the computational complexity to remain low. Publicly available, efficient implementation of IV-PSNR software was provided by the authors of this paper and is used by ISO/IEC MPEG for evaluation and research on the upcoming MPEG Immersive video (MIV) coding standard.

Free-viewpoint television (FTV) and virtual navigation appear to be hot research topics. In this paper, the authors study the practical development of free-viewpoint television systems that provide the functionality of virtual horizontal navigation around real scenes. The considerations are focused on practical systems that use purely optical depth estimation and might be employed in the next few years. The architectures of such systems are discussed in detail, including acquisition, preprocessing, depth estimation, compression, and presentation. In particular, the optimization of camera locations is discussed, and it is shown that video acquisition using camera pairs is advantageous for scenes with a substantial amount of occlusions. The theoretical considerations are supported by experimental results obtained for standard test multiview video sequences. Furthermore, the paper describes FTV video acquisition systems that consist of modules with pairs of cameras. The modules are sparsely located in arbitrary positions around a scene. Each camera module is equivalent to a video camera with a depth sensor. The hardware requirements, video processing algorithms, and experimental results are reported. In particular, for such systems, a compression technique is discussed that is more efficient than the new three-dimensional HEVC technology. The paper also describes new test video sequences that are obtained from the camera pairs sparsely distributed around scenes.

The MPEG immersive video (MIV) standard is the latest addition to the MPEG-I suite of standards. It focuses on the representation and coding of immersive media. MIV is designed to support virtual and extended reality applications that require six degrees of freedom visual interaction with the rendered scene. Edition-1 of MIV is now in its final phase of standardization. Leveraging conventional 2-D video codecs, the MIV standard efficiently codes volumetric scenes and allows advanced visual effects like bullet-time fly-throughs. The video feeds capturing the scene are first processed to identify a set of basic views that are augmented with additional information from all other views. The data are then intelligently packed into atlases and further compressed with any existing 2-D video codec of choice. Experimental results show BD-PSNR gains of up to 6 dB in the 10–20 Mbps range compared to a naive simulcast multiview video coding approach. This article concludes with an outlook on future extensions for the second edition of MIV.

This paper describes two methods for increasing the efficiency of the MPEG Immersive Video (MIV) coding standard. The methods manipulate the MIV-formatted atlas videos by considering the coding principles of the widely used video encoders, enhancing the encoding efficiency for the immersive video content. The first method, patch average color modification removes the constant component of all YCbCr components of each patch within atlases, resulting in the reduction of the number and magnitude of edges within a texture atlas video. The second proposed method changes the dynamic range of the geometry (depth) atlas, adapting to the quality of input depth maps. Both methods proposed by the authors of this paper were included into the Test Model for MPEG Immersive Video (TMIV), which is the reference implementation of the MIV codec. Moreover, the metadata syntax relating to the first proposed method was adopted to the ISO/IEC 23090-12 standard.

In this paper, we present a new objective quality metric designed for immersive video applications—IV-SSIM. The proposed IV-SSIM metric is an evolution of our previous work—IV-PSNR (immersive video peak signal-to-noise ratio)—which became a commonly used metric in research and ISO/IEC MPEG standardization activities on immersive video. IV-SSIM combines the advantages of IV-PSNR and metrics based on the structural similarity of images, being able to properly mimic the subjective quality perception of immersive video with its characteristic distortions induced by the reprojection of pixels between multiple views. The effectiveness of IV-SSIM was compared with 16 state-of-the-art quality metrics (including other metrics designed for immersive video). Tested metrics were evaluated in an immersive video coding scenario and against a commonly used image quality database—TID2013—showing their performance in both immersive and typical, non-immersive use cases. As presented, the proposed IV-SSIM metric clearly outperforms other metrics in immersive video applications, while also being highly competitive for 2D image quality assessment. The authors of this paper have provided a publicly accessible, efficient implementation of the proposed IV-SSIM metric, which is used by ISO/IEC MPEG video coding experts in the development of the forthcoming second edition of the MPEG immersive video (MIV) coding standard.