Hyperspectral imaging is a critical tool for gathering spatial-spectral information in various scientific research fields. As a result of improvements in spectral reconstruction algorithms, significant progress has been made in reconstructing hyperspectral images from commonly acquired RGB images. However, due to the limited input, reconstructing spectral information from RGB images is ill-posed. Furthermore, conventional camera color filter arrays (CFA) are designed for human perception and are not optimal for spectral reconstruction. To increase the diversity of wavelength encoding, we propose to place broadband encoding filters in front of the RGB camera. In this condition, the spectral sensitivity of the imaging system is determined by the filters and the camera itself. To achieve an optimal encoding scheme, we use an end-to-end optimization framework to automatically design the filters’ transmittance functions and optimize the weights of the spectral reconstruction network. Simulation experiments show that our proposed spectral reconstruction network has excellent spectral mapping capabilities. Additionally, our novel joint wavelength encoding imaging framework is superior to traditional RGB imaging systems. We develop the deeply learned filter and conduct actual shooting experiments. The spectral reconstruction results have an attractive spatial resolution and spectral accuracy.
Image pairs in under-illuminated scenes along with the presence of complex light sources often result in strong flare artifacts in images, affecting both image quality and the performance of downstream visual applications. Removing lens flare and ghosts is a challenging issue, particularly in low-light environments. Existing methods for flare removal are mainly restricted by inadequate simulation and real-world capture, resulting in singular categories of scattered flares and unavailable reflected ghosts. Therefore, a comprehensive deterioration procedure is crucial for generating a dataset for flare removal. We propose a methodology based on spatial position relationships for generating data pairs with flare deterioration, which is supported by theoretical analysis and real-world evaluation. Our procedure is comprehensive and realizes the similarity of scattered flares and the symmetric effect of reflected ghosts. We also construct a real-shot pipeline that respectively processes the effects of scattering and reflective flares, aiming to directly generate data for end-to-end methods. Experimental results demonstrate that our methodology adds diversity to existing flare datasets and constructs a comprehensive mapping procedure for flare data pairs. Our method facilitates the data-driven model to achieve better restoration in flare images and proposes a better evaluation system based on real shots, thus promoting progress in the area of real flare removal.
Nighttime image dehazing presents unique challenges due to the unevenly distributed haze caused by the color change of artificial light sources. This results in multiple interferences, including atmospheric light, glow, and direct light, which make the complex scattering haze interference difficult to accurately distinguish and remove. Additionally, obtaining pairs of high-definition data for fog removal at night is a difficult task. These challenges make nighttime image dehazing a particularly challenging problem to solve. To address these challenges, we introduced the haze scattering formula to more accurately express the haze in three-dimensional space. We also proposed a novel data synthesis method using the latest CG textures and lumen lighting technology to build scenes where various hazes can be seen clearly through ray tracing. We converted the complex 3D scattering relationship transformation into a 2D image dataset to better learn the mapping from 3D haze to 2D haze. Additionally, we improved the existing neural network and established a night haze intensity evaluation label based on the idea of optical PSF. This allowed us to adjust the haze intensity of the rendered dataset according to the intensity of the real haze image and improve the accuracy of dehazing. Our experiments showed that our data construction and network improvement achieved better visual effects, objective indicators, and calculation speed.
The large language model and high-level vision model have achieved impressive performance improvements with large datasets and model sizes. However, low-level computer vision tasks, such as image dehaze and blur removal, still rely on a small number of datasets and small-sized models, which generally leads to overfitting and local optima. Therefore, we propose a framework to integrate large-model prior into low-level computer vision tasks. Just as with the task of image segmentation, the degradation of haze is also texture-related. So we propose to detect gray-scale coding, network channel expansion, and pre-dehaze structures to integrate large-model prior knowledge into any low-level dehazing network. We demonstrate the effectiveness and applicability of large models in guiding low-level visual tasks through different datasets and algorithms comparison experiments. Finally, we demonstrate the effect of grayscale coding, network channel expansion, and recurrent network structures through ablation experiments. Under the conditions where additional data and training resources are not required, we successfully prove that the integration of large-model prior knowledge will improve the dehaze performance and save training time for low-level visual tasks.