Video Quality Assessment (VQA) for windowed six Degree of Freedom (windowed-6DoF) synthesized viewpoints is conducive to the development and broadcasting of 6DoF immersive video system. To explore this new field, we propose the subjective and objective VQA studies for windowed-6DoF synthesized videos. Stage one, a pioneering windowed-6DoF synthesized video quality database with compressed distortion, rendering distortion, and viewing path discomfort are presented, in which overall 128 videos are synthesized from four sequences with different distortion combinations. Stage two, a no-reference objective VQA method is proposed. Firstly, the video is converted into one traditional spatio-temporal domain (XY-T domain) and two new spatio-temporal domains (XT-Y and TY-X domains). Secondly, the emerging rendering distortion and viewing path discomfort are measured in the XT-Y and TY-X domains based on their respective characteristic statistical models, and the common compressed distortion is measured in XY-T domain. Thirdly, the image numbers of new domains are accurately determined through optical flow based depth estimation. Finally, the regression model plus weighting strategy is used to obtain the ultimate quality score of windowed-6DoF synthesized video. Experimental results on three synthesized video databases illustrate that the proposed objective NR VQA method has a good performance in evaluating various synthesized videos, compared with other classic and state-of-the-art 2D/3D IQA/VQA metrics.

Cell-phone use while driving results in potentially severe safety hazards. In this paper, a scheme for detecting cell-phone use that is based on deep learning is proposed, which can eliminate the potential risk by detecting the driver behavior and issuing an early warning. The proposed scheme consists of two stages: model training and practical testing. In the former, a multi-angle arrangement of cameras is first designed. Then, based on self-established data set, two independent convolutional neural networks (CNNs) are trained by optimizing the size and number of the convolution kernels, which can efficiently recognize cell-phones and hands in real time. In the testing stage, dynamic region extraction and skin color detection are employed as preprocessing to improve the accuracy of target recognition. Then, with the trained CNNs, the detection of cell-phone and hand targets is carried out, and the corresponding early warning is issued based on the distance of the interaction between the cell-phone and the hand. Numerous experiments are conducted and the results demonstrate that the proposed scheme can accurately detect cell-phone use during driving in real time, with a running time of 144 fps and an accuracy of 95.7%.

Due to sensor limitations, the light field (LF) images captured by the LF camera suffer from low dynamic range and are prone to poor exposure. To solve this problem, combining multi-exposure technology with LF camera imaging can achieve high dynamic range (HDR) LF imaging. However, for dynamic scenes, this approach tends to produce disturbing ghosting artifacts and destroy the parallax structure of the generated results. To this end, this paper proposes a novel ghost-free HDR LF imaging method using multi-attention learning and exposure guidance. Specifically, the proposed method first designs a multi-scale cross-attention module to achieve efficient multi-exposure LF feature alignment. After that, a dual self-attention-driven Transformer block is constructed to excavate the geometric information of LF and fuse the aligned LF features. In particular, exposure masks derived from middle-exposure are introduced in the feature fusion to guide the network to focus on information recovery in low- and high-brightness regions. Besides, a local compensation module is integrated to cope with local alignment errors and refine details. Finally, a multi-objective reconstruction strategy combined with exposure masks is employed to restore high-quality HDR LF images. Extensive experimental results on the benchmark dataset show that the proposed method generates HDR LF results with high spatial-angular quality consistency and outperforms the state-of-the-art methods in quantitative and qualitative comparisons. Furthermore, the proposed method can enhance the performance of existing LF applications, such as depth estimation.

Underwater images, as a crucial medium for storing ocean information in underwater sensors, play a vital role in various underwater tasks. However, they are prone to distortion due to the imaging environment, which leads to a decline in visual quality, which is an urgent issue for various marine vision systems to address. Therefore, it is necessary to develop underwater image enhancement (UIE) and corresponding quality assessment methods. At present, most underwater image quality assessment (UIQA) methods primarily rely on extracting handcrafted features that characterize degradation attributes, which struggle to measure complex mixed distortions and often exhibit discrepancies with human visual perception in practical applications. Furthermore, current UIQA methods lack the consideration of the perception perspective of enhanced effects. To this end, this paper employs luminance and saliency priors as critical visual information for the first time to measure the enhancement effect of global and local quality achieved by the UIE algorithms, named JLSAU. The proposed JLSAU is built upon an overall pyramid-structured backbone, supplemented by the Luminance Feature Extraction Module (LFEM) and Saliency Weight Learning Module (SWLM), which aim at obtaining perception features with luminance and saliency priors at multiple scales. The supplement of luminance priors aims to perceive visually sensitive global distortion of luminance, including histogram statistical features and grayscale features with positional information. The supplement of saliency priors aims to perceive visual information that reflects local quality variation both in spatial and channel domains. Finally, to effectively model the relationship among different levels of visual information contained in the multi-scale features, the Attention Feature Fusion Module (AFFM) is proposed. Experimental results on the public UIQE and UWIQA datasets demonstrate that the proposed JLSAU outperforms existing state-of-the-art UIQA methods.

The quality attenuation of synthesized video will directly affect the widespread adoption of immersive video, so it is crucial to design a quality assessment model that can determine whether the synthesized video meets the requirements of commercial broadcasting. However, designing a general-purpose no-reference quality assessment metric for synthesized videos is difficult due to the imperfect view synthesizing technology and scene diversity. Currently, the existed quality assessment algorithms for synthesized views are mostly based on handcrafted feature extraction. Inspired by the theory that the input stimuli are hierarchically and sparsely processed in the cerebral cortex, we combine Convolutional Neural Network (CNN) learning and sparse dictionary learning mechanisms, and propose a Multi-Model Learning based Blind Synthesized Video Quality Assessment (MML-BSVQA) metric. Firstly, to better reflect the spatio-temporal distortions, we convert the synthesized video into the Spatial Domain (SD), Vertical Temporal Domain (VTD) and Horizontal Temporal Domain (HTD) using video decomposition operation plus optical flow estimation. Secondly, we extract the deep semantic features from three domains based on a pre-trained CNN model. Thirdly, we represent the sparse features of three domains using respective trained over-complete sparse dictionaries. Note that both the CNN model and sparse dictionaries are trained on natural videos to ensure the general-purpose of the proposed MML-BSVQA metric. Finally, the score of a synthesized video is generated by weighted regression. Experimental results on three synthesized video databases demonstrate that the proposed metric outperforms classic and state-of-the-art quality assessment metrics.

Multiview video plus depth is one of the mainstream representations of 3D scenes in emerging free viewpoint video, which generates virtual 3D synthesized images through a depth-image-based-rendering (DIBR) technique. However, the inaccuracy of depth maps and imperfect DIBR techniques result in different geometric distortions that seriously deteriorate the users’ visual perception. An effective 3D synthesized image quality assessment (IQA) metric can simulate human visual perception and determine the application feasibility of the synthesized content. In this paper, a no-reference IQA metric based on visual-entropy-guided multi-layer features analysis for 3D synthesized images is proposed. According to the energy entropy, the geometric distortions are divided into two visual attention layers, namely, bottom-up layer and top-down layer. The feature of salient distortion is measured by regional proportion plus transition threshold on a bottom-up layer. In parallel, the key distribution regions of insignificant geometric distortion are extracted by a relative total variation model, and the features of these distortions are measured by the interaction of decentralized attention and concentrated attention on top-down layers. By integrating the features of both bottom-up and top-down layers, a more visually perceptive quality evaluation model is built. Experimental results show that the proposed method is superior to the state-of-the-art in assessing the quality of 3D synthesized images.