Reliable vision in challenging illumination conditions is one of the crucial requirements of future autonomous automotive systems. In the last decade, thermal cameras have become more easily accessible to a larger number of researchers. This has resulted in numerous studies which confirmed the benefits of the thermal cameras in limited visibility conditions. In this paper, we propose a learning-based method for visible and thermal image fusion that focuses on generating fused images with high visual similarity to regular truecolor (red-green-blue or RGB) images, while introducing new informative details in pedestrian regions. The goal is to create natural, intuitive images that would be more informative than a regular RGB camera to a human driver in challenging visibility conditions. The main novelty of this paper is the idea to rely on two types of objective functions for optimization: a similarity metric between the RGB input and the fused output to achieve natural image appearance; and an auxiliary pedestrian detection error to help defining relevant features of the human appearance and blending them into the output. We train a convolutional neural network using image samples from variable conditions (day and night) so that the network learns the appearance of humans in the different modalities and creates more robust results applicable in realistic situations. Our experiments show that the visibility of pedestrians is noticeably improved especially in dark regions and at night. Compared to existing methods we can better learn context and define fusion rules that focus on the pedestrian appearance, while that is not guaranteed with methods that focus on low-level image quality metrics.

In this letter, we present an efficient image denoising method combining quadtree-based nonlocal means (NLM) and locally adaptive principal component analysis. It exploits nonlocal multiscale self-similarity better, by creating sub-patches of different sizes using quadtree decomposition on each patch. To achieve spatially uniform denoising, we propose a local noise variance estimator combined with denoiser based on locally adaptive principal component analysis. Experimental results demonstrate that our proposed method achieves very competitive denoising performance compared with state-of-the-art denoising methods, even obtaining better visual perception at high noise levels.

Standardization bodies recommend various protocols to conduct subjective quality assessment (QA) of imaging systems. While many studies have compared these QA protocols, few have assessed the impact of different approaches for preprocessing and analyzing quality ratings. Furthermore, the effect of large versus small quality differences on the discrimination performance of protocols has not been extensively studied in video QA. This study examines these issues for four QA protocols. H.264 compressed medical videos and denoised natural scene videos were evaluated by expert and naive subjects. Scores were collected with four QA protocols-forced choice (FC), two ratio-scaled paired comparison methods: preference (Pref) and dissimilarity (Dissim), and single stimulus (SS)-and analyzed using combinations of different rating preprocessing approaches, generating a total of 14 methods. Performance metrics-probability and effect size-quantified the ability of the methods to discriminate between quality levels. The Pref and Dissim methods analyzed with classical multidimensional scaling and the SS method with Z-score transformation consistently outperformed the other methods. The type of preprocessing introduced large differences in the performance of individual protocols. Grouping stimuli pairs by small and large quality differences introduced significant differences in the performance rankings of the methods, with Pref and Dissim being most sensitive to small quality differences. For the future, we suggest further validation of the FC method, due to its simplicity and ease of use, and continued investigation into more robust raw scores transformation and statistical analysis methods for both SS and FC ratings.

Interpolation from a Color Filter Array (CFA) is the most common method for obtaining full color image data. Its success relies on the smart combination of a CFA and a demosaicing algorithm. Demosaicing on the one hand has been extensively studied. Algorithmic development in the past 20 years ranges from simple linear interpolation to modern neural-network-based (NN) approaches that encode the prior knowledge of millions of training images to fill in missing data in an inconspicious way. CFA design, on the other hand, is less well studied, although still recognized to strongly impact demosaicing performance. This is because demosaicing algorithms are typically limited to one particular CFA pattern, impeding straightforward CFA comparison. This is starting to change with newer classes of demosaicing that may be considered generic or CFA-agnostic. In this study, by comparing performance of two state-of-the-art generic algorithms, we evaluate the potential of modern CFA-demosaicing. We test the hypothesis that, with the increasing power of NN-based demosaicing, the influence of optimal CFA design on system performance decreases. This hypothesis is supported with the experimental results. Such a finding would herald the possibility of relaxing CFA requirements, providing more freedom in the CFA design choice and producing high-quality cameras.

Pedestrian detection in thermal and visible images is crucial for various applications, such as surveillance, driver assistance, and autonomous driving. In this paper, we propose a novel fusion scheme that effectively integrates multimodal features to improve detection performance. Our approach relies on Cross-Modality Reference Module (CMRM) for exchanging complementary features extracted from different modalities, solving incorrect sensor dominance problem in rare untrained-for contexts. We also utilize modality-specific region proposal networks to explore potential candidates separately in each modality, ensuring accurate and reliable proposals. The fusion of region proposals is performed using the Multimodal Fusion Module (MFM) that employs an attention mechanism to combine features based on their attention scores. To improve the robustness of the model in practical scenarios, we introduce a group of new data augmentation techniques, which simulate real-world challenges. Experimental evaluations conducted on the public KAIST, CVC-14, and FLIR datasets demonstrate the effectiveness of our proposed method. The results show that our fusion scheme significantly outperforms the existing methods in terms of detection performance by as much as 16.4% in practical scenarios.