This paper presents a novel approach to provide immersive free navigation with 6 Degrees of Freedom in real-time for natural and virtual scenery, for both static and dynamic content. Stemming from the state-of-the-art in Depth Image-Based Rendering and the OpenGL pipeline, this new View Synthesis method achieves free navigation at up to 90 FPS and can take any number of input views with their corresponding depth maps as priors. Video content can be played thanks to GPU decompression, supporting free navigation with full parallax in real-time. To render a novel viewpoint, each selected input view is warped using the camera pose and associated depth map, using an implicit 3D representation. The warped views are then blended all together to generate the chosen virtual view. Various view blending approaches specifically designed to avoid visual artifacts are compared. Using as few as four input views appears to be an optimal trade-off between computation time and quality, allowing to synthesize high-quality stereoscopic views in real-time, offering a genuine immersive virtual reality experience. Additionally, the proposed approach provides high-quality rendering of a 3D scenery on holographic light field displays. Our results are comparable - objectively and subjectively - to the state of the art view synthesis tools NeRF and LLFF, while maintaining an overall lower complexity and real-time rendering.
Reaction-diffusion models have been proposed for decades to capture the growth of gliomas, the most common primary brain tumors. However, ill-posedness of the initialization at diagnosis time and parameter estimation of such models have restrained their clinical use as a personalized predictive tool. In this work, we investigate the ability of deep convolutional neural networks (DCNNs) to address commonly encountered pitfalls in the field. Based on 1200 synthetic tumors grown over real brain geometries derived from magnetic resonance (MR) data of six healthy subjects, we demonstrate the ability of DCNNs to reconstruct a whole tumor cell-density distribution from only two imaging contours at a single time point. With an additional imaging contour extracted at a prior time point, we also demonstrate the ability of DCNNs to accurately estimate the individual diffusivity and proliferation parameters of the model. From this knowledge, the spatio-temporal evolution of the tumor cell-density distribution at later time points can ultimately be precisely captured using the model. We finally show the applicability of our approach to MR data of a real glioblastoma patient. This approach may open the perspective of a clinical application of reaction-diffusion growth models for tumor prognosis and treatment planning.
Depth Image-Based Rendering (DIBR) and 6 Degrees-of-Freedom (6DoF) navigation are actually very active fields for immersive video, where the availability of common test conditions is mandatory to compare results. For this purpose, we propose different datasets to evaluate a wide range of tools needed by free navigation: a high-density HD image set, composed of 80k camera positions covering the entire 3D space, necessary to simulate all the head rotations and translations of a common user, and a UHD video set, composed of two sequences acquired with 3 by 5 cameras array. All the processing pipeline to obtain quality results is also detailed.
We present a novel, first-of-its-kind view synthesis method for plenoptic images, which enables the direct manipulation of images in the micro-images array format, thereby bypassing intermediate transformation steps. Current plenoptic imaging approaches typically rely on an initial conversion to dense multiview images, also known as subaperture images extraction. However, the use of subaperture images presents two main limitations that ultimately impact further processing. First, existing subaperture view extraction methods offer limited control over camera parameters, resolutions, and poses of the subaperture views, which are also constrained to a small area around the main lens, thus restricting free navigation. Second, subaperture images are susceptible to artifacts which can propagate to subsequent processes such as calibration, depth estimation and view synthesis. In this paper, we propose a camera model that enables depth image-based rendering with plenoptic cameras, in a way that allows for the direct synthesis of any target viewpoint. In our evaluation, we show that our method expands view synthesis extrapolation to a range that is two to three times greater than that of pipelines requiring a conversion to subaperture images, including generally accepted tools such as depth image-based rendering and learning-based rendering approaches.
6 January 2026 in International Journal of Computer Vision
We introduce a structure-from-motion method specifically designed to process the raw micro-images captured by plenoptic 2.0 cameras. Unlike traditional monocular cameras, plenoptic cameras incorporate a micro-lens array between the sensor and the main lens, capturing depth information at the expense of a more complex set of parameters to evaluate. Instead of simply integrating their projection model into the classical structure-from-motion pipeline, our contribution identifies the pinhole cameras-driven constraints and takes advantage of the inherent disparity information present in plenoptic cameras. This facilitates a robust initialization of the reconstruction. Our method shortcuts two of the limitations of the classical structure-from-motion: the ambiguity found in scenes captured with low angular disparity and the scale ambiguity. It enables the reconstruction of scenes captured by multiple uncalibrated plenoptic cameras, without using any calibration pattern or subaperture view extraction step. Our method undergoes experimental validation on both natural and synthetic datasets, showing a 10% error accuracy for relative pose estimation, which is comparable to calibration-pattern based methods. The results are robust to coarse initialization. Contrary to classical structure-from-motion, it is able to reconstruct scenes with parallel facing cameras. It also shows greater accuracy than reconstruction methods based on pinhole camera conversion.