This paper introduces an end-to-end learned image compression system, termed ANFIC, based on Augmented Normalizing Flows (ANF). ANF is a new type of flow model, which stacks multiple variational autoencoders (VAE) for greater model expressiveness. The VAE-based image compression has gone mainstream, showing promising compression performance. Our work presents the first attempt to leverage VAE-based compression in a flow-based framework. ANFIC advances further compression efficiency by stacking and extending hierarchically multiple VAE’s. The invertibility of ANF, together with our training strategies, enables ANFIC to support a wide range of quality levels without changing the encoding and decoding networks. Extensive experimental results show that in terms of PSNR-RGB, ANFIC performs comparably to or better than the state-of-the-art learned image compression. Moreover, it performs close to VVC intra coding, from low-rate compression up to perceptually lossless compression. In particular, ANFIC achieves the state-of-the-art performance, when extended with conditional convolution for variable rate compression with a single model. The source code of ANFIC can be found at https://github.com/dororojames/ANFIC .

Free-viewpoint television (FTV) and virtual navigation appear to be hot research topics. In this paper, the authors study the practical development of free-viewpoint television systems that provide the functionality of virtual horizontal navigation around real scenes. The considerations are focused on practical systems that use purely optical depth estimation and might be employed in the next few years. The architectures of such systems are discussed in detail, including acquisition, preprocessing, depth estimation, compression, and presentation. In particular, the optimization of camera locations is discussed, and it is shown that video acquisition using camera pairs is advantageous for scenes with a substantial amount of occlusions. The theoretical considerations are supported by experimental results obtained for standard test multiview video sequences. Furthermore, the paper describes FTV video acquisition systems that consist of modules with pairs of cameras. The modules are sparsely located in arbitrary positions around a scene. Each camera module is equivalent to a video camera with a depth sensor. The hardware requirements, video processing algorithms, and experimental results are reported. In particular, for such systems, a compression technique is discussed that is more efficient than the new three-dimensional HEVC technology. The paper also describes new test video sequences that are obtained from the camera pairs sparsely distributed around scenes.

We propose a new coding technology for 3D video represented by multiple views and the respective depth maps. The proposed technology is demonstrated as an extension of the recently developed high efficiency video coding (HEVC). One base views are compressed into a standard bitstream (like in HEVC). The remaining views and the depth maps are compressed using new coding tools that mostly rely on view synthesis. In the decoder, those views and the depth maps are derived via synthesis in the 3D space from the decoded baseview and from data corresponding to small disoccluded regions. The shapes and locations of those disoccluded regions can be derived by the decoder without any side information transmitted. To achieve high compression efficiency, we propose several new tools such as depth-based motion prediction, joint high frequency layer coding, consistent depth representation, and nonlinear depth representation. The experiments show high compression efficiency of the proposed technology. The bitrate needed for transmission of two side views with depth maps is mostly less than 50% than that of the bitrate for a single-view video.

In this paper we consider immersive visual media that are currently researched within scientific community. These include the well-established technologies, like 360-degree panoramas, as well as those being intensively developed, like free viewpoint video and point-cloud based systems. By the use of characteristic examples, we define the features of the immersive visual media that distinguish them from the classical 2D video. Also, we present the representation technologies that are currently considered by the scientific community, especially in the context of standardization of the immersive visual media in the MPEG-I project recently launched by ISO/IEC.

The paper presents a new method of depth estimation, dedicated for free-viewpoint television (FTV) and virtual navigation (VN). In this method, multiple arbitrarily positioned input views are simultaneously used to produce depth maps characterized by high inter-view and temporal consistencies. The estimation is performed for segments and their size is used to control the trade-off between the quality of depth maps and the processing time of depth estimation. Additionally, an original technique is proposed for the improvement of temporal consistency of depth maps. This technique uses the temporal prediction of depth, thus depth is estimated for P-type depth frames. For such depth frames, temporal consistency is high, whereas estimation complexity is relatively low. Similarly, as for video coding, I-type depth frames with no temporal depth prediction are used in order to achieve robustness. Moreover, we propose a novel parallelization technique that significantly reduces the estimation time. The method is implemented in C++ software that is provided together with this paper, so other researchers may use it as a new reference for their future works. In performed experiments, MPEG methodology was used whenever possible. The provided results demonstrate the advantages over the Depth Estimation Reference Software (DERS) developed by MPEG. The fidelity of a depth map, measured by the quality of synthesized views, is higher on average by 2.6 dB. This significant quality improvement is obtained despite a significant reduction of the estimation time, on average 4.5 times. The application of the proposed temporal consistency enhancement method increases this reduction to 29 times. Moreover, the proposed parallelization results in the reduction of the estimation time up to 130 times (using 6 threads). As there is no commonly accepted measure of the consistency of depth maps, the application of compression efficiency of depth is proposed as a measure of depth consistency.