4 March 2022 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Lensless imaging has emerged as a potential solution towards realizing ultra-miniature cameras by eschewing the bulky lens in a traditional camera. Without a focusing lens, the lensless cameras rely on computational algorithms to recover the scenes from multiplexed measurements. However, the current iterative-optimization-based reconstruction algorithms produce noisier and perceptually poorer images. In this work, we propose a non-iterative deep learning-based reconstruction approach that results in orders of magnitude improvement in image quality for lensless reconstructions. Our approach, called FlatNet , lays down a framework for reconstructing high-quality photorealistic images from mask-based lensless cameras, where the camera's forward model formulation is known. FlatNet consists of two stages: (1) an inversion stage that maps the measurement into a space of intermediate reconstruction by learning parameters within the forward model formulation, and (2) a perceptual enhancement stage that improves the perceptual quality of this intermediate reconstruction. These stages are trained together in an end-to-end manner. We show high-quality reconstructions by performing extensive experiments on real and challenging scenes using two different types of lensless prototypes: one which uses a separable forward model and another, which uses a more general non-separable cropped-convolution model. Our end-to-end approach is fast, produces photorealistic reconstructions, and is easy to adopt for other mask-based lensless cameras.
23 December 2020 in IEEE Transactions on Image Processing
Light Field (LF) offers unique advantages such as post-capture refocusing and depth estimation, but low-light conditions severely limit these capabilities. To restore low-light LFs we should harness the geometric cues present in different LF views, which is not possible using single-frame low-light enhancement techniques. We propose a deep neural network L3Fnet for Low-Light Light Field (L3F) restoration, which not only performs visual enhancement of each LF view but also preserves the epipolar geometry across views. We achieve this by adopting a two-stage architecture for L3Fnet. Stage-I looks at all the LF views to encode the LF geometry. This encoded information is then used in Stage-II to reconstruct each LF view. To facilitate learning-based techniques for low-light LF imaging, we collected a comprehensive LF dataset of various scenes. For each scene, we captured four LFs, one with near-optimal exposure and ISO settings and the others at different levels of low-light conditions varying from low to extreme low-light settings. The effectiveness of the proposed L3Fnet is supported by both visual and numerical comparisons on this dataset. To further analyze the performance of low-light restoration methods, we also propose the L3F-wild dataset that contains LF captured late at night with almost zero lux values. No ground truth is available in this dataset. To perform well on the L3F-wild dataset, any method must adapt to the light level of the captured scene. To do this we use a pre-processing block that makes L3Fnet robust to various degrees of low-light conditions. Lastly, we show that L3Fnet can also be used for low-light enhancement of single-frame images, despite it being engineered for LF data. We do so by converting the single-frame DSLR image into a form suitable to L3Fnet, which we call as pseudo-LF . Our code and dataset is available for download at https://mohitlamba94.github.io/L3Fnet/
2 December 2021 in IEEE Transactions on Image Processing
Thermography is a useful imaging technique as it works well in poor visibility conditions. High-resolution thermal imaging sensors are usually expensive and this limits the general applicability of such imaging systems. Many thermal cameras are accompanied by a high-resolution visible-range camera, which can be used as a guide to super-resolve the low-resolution thermal images. However, the thermal and visible images form a stereo pair and the difference in their spectral range makes it very challenging to pixel-wise align the two images. The existing guided super-resolution (GSR) methods are based on aligned image pairs and hence are not appropriate for this task. In this paper, we attempt to remove the necessity of pixel-to-pixel alignment for GSR by proposing two models: the first one employs a correlation-based feature-alignment loss to reduce the misalignment in the feature-space itself and the second model includes a misalignment-map estimation block as a part of an end-to-end framework that adequately aligns the input images for performing guided super-resolution. We conduct multiple experiments to compare our methods with existing state-of-the-art single and guided super-resolution techniques and show that our models are better suited for the task of unaligned guided super-resolution from very low-resolution thermal images.