Occlusions will reduce the performance of systems in many computer vision applications with discontinuous surfaces of 3D scenes. We explore a signal-processing framework of occlusions based on the light ray visibility to improve the rendering quality of views. An occlusion field (OCF) theory is derived by calculating the relationship between the occluded light rays and the nonoccluded light rays to quantify the occlusion degree (OCD). The OCF framework can describe the various in-scene information captured by the changes in the camera configuration (i.e., position and direction) through a quantitative description of the occlusion information. From a spectral analysis of the OCF, we mathematically derive analytical functions to determine the changing relationship between the scene and the camera configuration. A reconstruction filter can be designed to achieve interference cancellation and compensate for the missing information caused by the occlusions. Our measurements of different occlusions using this OCF framework included both synthetic and actual scenes. The experimental results show that the proposed OCF framework can improves the rendering quality of views and outperforms other known occlusion quantization schemes in a complex scene.

Light field sampling (LFS) theory can properly reduce minimum sampling rate while ensuring that novel views are not distorted for image-based rendering (IBR). The minimum sampling rate is determined by spectral support of light field. The spectral support of light field has studied the influence of the following factors: the minimum depth and the maximum depth, non-Lambertian reflections, whether the scene surfaces are flat, maximum frequency of painted signals. In this paper, we further perfect the light field spectrum analysis from the quantitative description of scene texture information based on the existing spectrum analysis theory. The quantification of texture information can be interactively refined via detected regional entropy. Thus, we can derive a spectral analytical function of light field with respect to texture information. The new function allows the spectral support of light field to be analyzed and estimated for different texture information associated with scene objects. In this way, we limit the spectral analysis problems of light field to those of a simpler signal. We show that this spectral analysis approach can be easily extended to arbitrary scene complexity levels, as we simplify the LFS of complex scenes to a plane. Additionally, the spectral support of light field broadens as the plane texture information becomes more complex. We present experimental results to demonstrate the performance of LFS with texture information, verify our theoretical analysis, and extend our conclusions on the optimal minimum sampling rate.

Occlusions are common phenomena in light field rendering (LFR) technology applications. The 3-D spatial structures of some features may be missing or incorrect when capturing some samples due to occlusion discontinuities. Most prior works on LFR, however, have neglected occlusions from other objects in 3-D scenes that do not participate in the capturing and rendering of the light field. To improve rendering quality, this report proposes an occlusion probability learning framework (OPLF) based on a deep Boltzmann machine (DBM) to compensate for the occluded information. In the OPLF, an occlusion probability density model is applied to calculate the visibility scores, which are modeled as hidden variables. Additionally, the probability of occlusion is related to the visibility, the camera configuration (i.e., position and direction), and the relationship between the occlusion object and occluded object. Furthermore, a deep probability model based on the OPLF is used for learning the occlusion relationship between the camera and object in multiple layers. The proposed OPLF can optimize the LFR quality. Finally, to verify the claimed performance, we also compare the OPLF with the most advanced occlusion theory and light field reconstruction algorithms. The experimental results show that the proposed OPLF outperforms other known occlusion quantization schemes.

The frequency estimation of the plenoptic function (POF) is an important research topic in spectral analysis for determining the minimum sampling rate of image-based rendering. In this paper, we mathematically derive a frequency estimation function of the POF using the autocorrelation theory. The autocorrelation function (ACF) of the POF is studied along both the spatial and image plane frequency axes. The influence of the scene's complexity and depth on the ACF of the POF is analyzed. Furthermore, we study the frequency estimation error to analyze the performance of the method. Existing techniques typically use Fourier transformation to determine the frequency of the POF. The technique presented herein simply starts from measurements of the light field in a finite number of positions, and the information from the light field is analyzed in the spatial domain. Finally, experimental results are presented to demonstrate that the proposed method can effectively estimate the frequency of the POF. The results are shown to be in good agreement with traditional methods.

Rendering quality optimization theory is one of the most basic and fascinating components of image-based rendering (IBR). The rendering quality of virtual views is related to the information in the source images. Capturing the image information depends on the geometric configuration of the camera (GCC) and, in particular, the positions and shooting directions of the cameras. Therefore, the rendering quality of virtual views can be improved by optimizing the GCC. This paper investigates the relationship between the GCC and the geometric configurations of the virtual view (GCVV). The influence of the GCC on the rendering quality is also analyzed. Based on the relationships among the GCC, GCVV and rendering quality, a mathematical model of the non-coverage field (NCF) is proposed to quantify the rendering quality. The performance of the NCF is also analyzed using a set of GCVVs and GCCs. Furthermore, an optimization algorithm based on the NCF that simultaneously optimizes the position and direction of the GCC is presented. The proposed technique can be applied to obtain the optimal rendering quality of IBR for the linear case of camera positions and virtual views. Finally, experimental results are presented and compared with the theoretical results.