25 June 2021 in IEEE Transactions on Image Processing
Light Field (LF) cameras are considered to have many potential applications since angular and spatial information is captured simultaneously. However, the limited spatial resolution has brought lots of difficulties in developing related applications and becomes the main bottleneck of LF cameras. In this paper, an end-to-end learning-based method is proposed to simultaneously reconstruct all view images in LFs with higher spatial resolution. Based on the epipolar geometry, view images in one LF are first grouped into several image stacks and fed into different network branches to learn sub-pixel details for each view image. Since LFs have dense sampling in angular domain, sub-pixel details in multiple spatial directions are learned from corresponding angular directions in multiple branches, respectively. Then, sub-pixel details from different directions are further integrated to generate global high-frequency residual details. Combined with the spatially upsampled LF, the final LF with high spatial resolution is obtained. Experimental results on synthetic and real-world datasets demonstrate that the proposed method outperforms other state-of-the-art methods in both visual and numerical evaluations. We also implement the proposed method on LFs with different angular resolution and experiments show that the proposed method achieves superior results than others, especially for LFs with small angular resolution. Furthermore, since the epipolar geometry is fully considered, the proposed network shows good performances in preserving the inherent epipolar property in LF images.
26 July 2021 in IEEE Transactions on Computational Imaging
Due to hardware restriction, it is costly to capture densely-sampled Light Fields (LFs) with high angular and spatial resolution, which becomes the main bottleneck of LFs development. In this paper, we propose a learning-based framework to synthesize novel views and reconstruct densely-sampled LFs from sparsely-sampled LFs. In the proposed framework, micro-lens image stacks and view image stacks are separately grouped, in which details in novel views are explored from spatial and angular domain. The two kinds of stacks contain epipolar information and 3D convolution layers are employed to effectively extract features that include structure information. Moreover, an innovative way is proposed to synthesize views by upsampling micro-lens image stacks using deconvolution layers. The parameters in decovolution layers provide the view position information and different interpolation and extrapolation tasks can be explicitly modeled. It is validated that this view synthesis module can be embedded in different frameworks and improve the related performances. Without precise depth estimation and view warping, the proposed method is mainly designed for reconstructing LFs with small baselines. Related experimental results show that the proposed model outperforms other state-of-the-art methods in terms of both visual and numerical evaluations. Furthermore, the consistency between synthesized views and the intrinsic structure information is well preserved in the proposed method.
26 April 2022 in IEEE Transactions on Computational Imaging
Limited by sensor resolution, Light Field (LF) images often suffer from an inherent trade-off between angular resolution and spatial resolution. LF reconstruction, including Spatial Super-Resolution (LFSSR) and Angular Super-Resolution (LFASR), upsamples LF in spatial and angular domain based on the complementary information in different views. In this paper, we propose efficient pseudo-4D end-to-end frameworks for LFSSR and LFASR, respectively. Since the epipolar information and spatial-angular information reflect the relationship between views, we propose to fully consider both of them to exploit complementary information thoroughly. Specifically, we first extract epipolar information from horizontally and vertically stacked views separately. Then an Epipolar-Aware Grid (EAG) network composed of Dual Interactive Blocks (DIBs) fully interacts the epipolar information, characterizing the grid-like LF parallax structure. In order to effectively achieve dense interaction between the spatial and angular domain to extract complementary information, several Parallel Spatial-Angular Integration Blocks (PSAIBs) are further introduced. For LFs with large baselines, we also propose a generic shear attention network, which generalizes the model designed for small baselines to large baselines without depth estimation. As compared to the state-of-the-art methods, extensive experiments conducted on both synthetic and real-world datasets demonstrate our superiority in LFSSR tasks and LFASR tasks.
16 April 2025 in IEEE Transactions on Visualization and Computer Graphics
Recently, Light Field (LF) shows great potential in removing occlusion since the objects occluded in some views may be visible in other views. However, existing LF-based methods implicitly model each scene and can only remove objects that have positive disparities in one central views. In this article, we propose a novel Progressive Multi-Plane Images (MPI) Construction method specifically designed for LF-based occlusion removal. Different from the previous MPI construction methods, we progressively construct MPIs layer by layer in order from near to far. In order to accurately model the current layer, the positions of foreground occlusions in the nearer layers are taken as occlusion prior. Specifically, we propose an Occlusion-Aware Attention Network to generate each layer of MPIs with reliable information in occluded regions. For each layer, occlusions in the current layer are filtered out so that the background is better recovered just using the visible views instead of the other occluded views. Then, by simply removing the layers containing occlusions and rendering MPIs in kinds of viewpoints, the occlusion removal results for different views are generated. Experiments on synthetic and real-world scenes show that our method outperforms state-of-the-art LF occlusion removal methods in quantitative and visual comparisons. Moreover, we also apply the proposed progressive MPI construction method to the view synthesis task. The occlusion edges in our synthesized views achieve significantly better quality, which also verifies that our method can better model the occluded regions.
27 December 2024 in IEEE Transactions on Multimedia
As densely-sampled Light Field (LF) images are beneficial to many applications, LF reconstruction becomes an important technology in related fields. Recently, neural rendering shows great potential in reconstruction tasks. However, volume rendering in existing methods needs to sample many points on the whole camera ray or epipolar line, which is time-consuming. In this paper, specifically for LF images with regular angular sampling, we propose a novel Structure-Aware Pre-Selected neural rendering framework for LF reconstruction. Instead of sampling on the whole epipolar line, we propose to sample on several specific positions, which are estimated using the color and inherent scene structure information explored in the regular angular sampled LF images. Sampling only a few points that closely match the target pixel, the feature of the target pixel is quickly rendered with high-quality. Finally, we fuse the features and decode them in the view dimension to obtain the final target view. Experiments show that the proposed method outperforms the state-of-the-art LF reconstruction methods in both qualitative and quantitative comparisons across various tasks. Our method also surpasses the most existing methods in terms of speed. Moreover, without any retraining or fine-tuning, the performance of our method with no-per-scene optimization is even better than the methods with per-scene optimization.