Infrared image retains typical thermal targets while visible image preserves rich texture details, image fusion aims to reconstruct a synthesized image containing prominent targets and abundant texture details. Most of deep learning-based methods mainly focus on convolution operation to extract the local features, but do not fully consider their multi-scale characteristics and global dependencies, which may cause loss of target regions and texture details in the fused image. Towards this goal, we present a unified multi-scale densely connected fusion network in this paper, named as UNFusion. We carefully design a multi-scale encoder-decoder architecture that can efficiently extract and reconstruct multi-scale deep features. Dense skip connections are employed in both encoder and decoder sub-networks to reuse all the intermediate features of different layers and scales for fusion tasks. In the fusion layer, $L_{p} $ normalized attention models, which include three kinds of different norms, are proposed to highlight and combine these deep features from spatial and channel dimensions, and the combined spatial and channel attention maps are used to reconstruct a final fused image. We conduct extensive experiments on the public TNO and Roadscene datasets, and the results demonstrate that our UNFusion can simultaneously preserve high brightness of typical thermal targets and abundant texture details to obtain superior scene representation and better visual perception. Besides, our UNFusion achieves better fusion performance and transcends other state-of-the-art methods in terms of qualitative and quantitative comparisons. Our code is available at https://github.com/Zhishe-Wang/UNFusion .

The computational modeling of human visual attention has received much attention in recent decades. In advanced industrial applications, it has been demonstrated that computational visual attention models (CVAMs) can predict visual attention very similarly to human visual attention. However, it is controversial whether the driver’s eye fixation location (EFL) or the predicted eye fixation location of computational visual attention models is more reliable and helpful for actual driving. To address this issue, an open database of videos taken under the most common 18 driving conditions in everyday driving has been established. In experiments using this database, expert drivers found that it was not sufficient for drivers to rely on only one of the two EFLs. Based on this finding, a hybrid EFL recommendation strategy is proposed for improving driving safety. By extracting visual characteristics from human dynamic vision, the performance of the proposed recommendation method demonstrates its potential value in these collected driving tasks. In addition, the visual comfort of driving is further addressed to enhance the safety of driving. From the results of experiments on 108 driving video clips taken of the most common 18 real driving conditions, it is confirmed that the proposed EFL recommendation achieves an experience rating of driving comfort between 88.1 and 92.7 out of 100.

It is of great significance for safe driving to study drivers' eye movement and driving operation behavior when they encounter other road users violating traffic rules. The underlying reason is that most drivers are unable to process the unexpected visual stimulation, which is more likely to lead to driving accidents, especially in a hybrid situation. In this study, a driving simulator is used to design driving scenarios and study the driving performance of drivers with different driving experiences when other road users violate traffic rules. The experimental results show that some novice drivers ignore the position of their own vehicle when they encounter traffic violations which will lead to the collision with other road users. Moreover, some novice drivers can only perform one of the operations between steering and braking to avoid collision in these emergent situations. They cannot reasonably combine braking and steering to complete emergency driving operations like experienced drivers. Finally, when the driving difficulty increases, experienced drivers spend less time looking and more time scanning their surroundings to ensure that they can cope with the more complex driving environment while novice drivers do the opposite. This study reveals the difference between novices and experienced drivers, which paves a useful reference for the future advanced driving assistance system.

Driver identification has been popular in the field of driving behavior analysis, which has a broad range of applications in anti-thief, driving style recognition, insurance strategy, and fleet management. However, most studies to date have only researched driver identification without a robust verification stage. This paper addresses driver identification and verification through a deep learning (DL) approach using psychological behavioral data, i.e., vehicle control operation data and eye movement data collected from a driving simulator and an eye tracker, respectively. We design an architecture that analyzes the segmentation windows of three-second data to capture unique driving characteristics and then differentiate drivers on that basis. The proposed model includes a fully convolutional network (FCN) and a squeeze-and-excitation (SE) block. Experimental results were obtained from 24 human participants driving in 12 different scenarios. The proposed driver identification system achieves an accuracy of 99.60% out of 15 drivers. To tackle driver verification, we combine the proposed architecture and a Siamese neural network, and then map all behavioral data into two embedding layers for similarity computation. The identification system achieves significant performance with average precision of 96.91%, recall of 95.80%, F1 score of 96.29%, and accuracy of 96.39%, respectively. Importantly, we scale out the verification system to imposter detection and achieve an average verification accuracy of 90.91%. These results imply the invariable characteristics from human factors rather than other traditional resources, which provides a superior solution for driving behavior authentication systems.

Traditional fusion methods based on deep learning mainly employ convolutional or self-attention operations to model local or global dependencies, which often lead to the oversight of frequency-domain information. To address this deficiency, we introduce a unified frequency adversarial learning network, termed FreqGAN. Our method involves a frequency-compensated generator that employs discrete wavelet transformation to decompose encoded spatial features into multiple frequency bands. Leveraging skip connections, low and high-frequency components are respectively directed into the encoder and decoder, compensating for additional outline and detail. Moreover, we construct a hybrid frequency aggregation module, which enables a progressive optimization of activity levels across multiple scales and makes the various frequency bands correlated. Complementing our generative model, we devise dual frequency-constrained discriminators. These discriminators are tasked with dynamically adjusting weights for each input frequency band, thereby obligating the generator to accurately reconstruct salient frequency information from different modality images. Additionally, a frequency-supervised function is formulated to further safeguard against the loss of frequency information. Our comprehensive experimental evaluations, encompassing a wide range of fusion tasks and subsequent applications, distinctly highlight FreqGAN’s superior performance, establishing it as a frontrunner in comparison to existing state-of-the-art alternatives. The source codes are forthcoming at: https://github.com/Zhishe-Wang/FreqGAN.

Recent existing methods generally adopt a simple concatenation or addition strategy to integrate features at the fusion layer, failing to adequately consider the intrinsic characteristics of different modal images and feature interaction of different scales, which may produce a limited fusion performance. Toward this end, we introduce a cross-scale iterative attentional adversarial fusion network, namely CrossFuse. More specifically, in the generator, we design a cross-modal attention integrated module to merge the intrinsic content of different modal images. The parallel spatial-independent and channel-independent pathways are proposed to calculate the attentional weights, which are assigned to measure the activity levels of source images at the same scale. Moreover, we construct a cross-scale iterative decoder framework to interact with different modality features at different scales, which can constantly optimize their activity levels. By this means, the generator learns to integrate their modality characteristics via attentional weights in an iterative manner, and the generated result characterizes competitive infrared radiant intensity and distinct visible detail description. Extensive experiments on three different benchmarks demonstrate that our CrossFuse outperforms other nine state-of-the-art methods in terms of fusion performance, generalization ability and computational efficiency. Our codes will be released at https://github.com/Zhishe-Wang/CrossFuse.