Due to the limitation of commercial light field camera hardware devices, the imaging field of view is quite narrow. Numerous Light Field Image (LFI) stitching algorithms have been developed to expand the field of view. However, it is highly challenging to compare the performance of LFI stitching algorithms in a fair manner. Currently, due to the absence of a comprehensive benchmark database for subjective rating and a reliable objective quality metric, it is fairly difficult to comprehensively and accurately compare the actual performance of existing LFI stitching algorithms. In this study, we dedicate our efforts to the development of quality metrics for stitched Wide field of view LFI (WLFI) from subjective and objective assessment aspects. Specifically, we build the first stitched WLFI database, which provides the stitched WLFIs generated by eight representative LFI stitching algorithms, along with their corresponding subjective rating scores. Secondly, an effective blind stitched WLFI quality metric is developed to accurately assess the visual quality degradation. Extensive experiments conducted over our established WLFI database demonstrate that the proposed metric achieves higher consistency with subjective ratings than the competing quality metrics.

Spontaneous speech emotion recognition is a new and challenging research topic. In this paper, we propose a new method of spontaneous speech emotion recognition on the basis of binaural representations and deep convolutional neural networks (CNNs). The proposed method initially employs multiple CNNs to learn deep segment-level binaural representations such as Left-Right and Mid-Side pairs from the extracted image-like Mel-spectrograms. These CNNs are fine-tuned on target emotional speech datasets from a pre-trained image CNN model. Then, a new feature pooling strategy, called block-based temporal feature pooling, is proposed to aggregate the learned segment-level features for producing fixedlength utterance-level features. Based on the utterance-level features, linear support vector machines (SVM) is adopted for emotion classification. Finally, a two-stage score-level fusion strategy is used to integrate the obtained results from Left-Right and Mid-Side pairs. Extensive experiments on two challenging spontaneous emotional speech datasets, including the AFEW5.0 and BAUM-1s databases, demonstrate the effectiveness of our proposed method.

Light field imaging captures both the intensity and directional information of light rays, providing users with more immersive visual experience. However, during the processes of imaging, processing, coding and reconstruction, light field images (LFIs) may encounter various distortions that degrade their visual quality. Compared to two-dimensional image quality assessment, light field image quality assessment (LFIQA) needs to consider not only the image quality in the spatial domain but also the quality degradation in the angular domain. To effectively model the factors related to visual perception and LFI quality, this paper proposes a multi-scale attention feature fusion based blind LFIQA metric, named MAFBLiF. The proposed metric consists of the following parts: MLI-Patch generation, spatial-angular feature separation module, spatial-angular feature extraction backbone network, pyramid feature alignment module and patch attention module. These modules are specifically designed to extract spatial and angular information of LFIs, and capture multi-level information and regions of interest. Furthermore, a pooling scheme guided by the LFI’s gradient information and saliency is proposed, which integrates the quality of all MLI-patches into the overall quality of the input LFI. Finally, to demonstrate the effectiveness of the proposed metric, extensive experiments are conducted on three representative LFI quality evaluation datasets. The experimental results show that the proposed metric outperforms other state-of-the-art image quality assessment metrics. The code will be publicly available at https://github.com/oldblackfish/MAFBLiF.

Tone-mapping technology aims to transform high dynamic range (HDR) images into standard dynamic range images for visualizing on standard screens. However, various visual artifacts will inevitably be induced during the tone-mapping process. To quantify the visual quality degradation for tone-mapped images (TMIs) accurately and automatically, a robust blind TMI quality measurement method is developed by the analysis of low-level and high-level perceptual characteristics. Specifically, for low-level visual features, considering that the tone-mapping operator is prone to destroy the local image contrast, which conveys vital structural information, we resort to the joint statistical features based on gradient map and Laplacian of Gaussian response to portray the local image contrast variation in TMIs. In addition, chromatic local binary pattern (CLBP) is leveraged to measure the colorfulness degradation over four chromatic descriptor maps. For high-level quality-aware features, deep-learned features are utilized by a pretrained convolutional neural network to characterize the semantics variation. Finally, the extracted low-level and high-level features are combined and mapped into an overall quality score by regression function. Extensive experiments show that our metric achieves a higher performance over two publicly available TMI benchmark databases.

Image stitching technology aims to generate a Stitched Panoramic Image (SPI) by stitching multiple narrow-view images containing over-lapping regions. However, heterogeneous artifacts may be introduced during the process of image stitching, which affects the visual perception. To automatically and accurately evaluate the perceptual quality of SPIs, a novel local visual and global deep features based blind SPI quality evaluation method is proposed in this paper. To be specific, with the consideration that image stitching mainly destroys structure, texture and color information, we first design a color structure-texture joint dictionary trained from the constructed stitched-specific image patches dataset. Given an input SPI, its local visual and global deep features are extracted to characterize the stitched-specific distortions. For local visual features, the trained dictionary is employed to capture structure, texture and color distortions by sparse features extraction. Then, considering that sparse features are insensitive to weak structural distortions, weighted local binary pattern features are extracted to measure various weak distortions. For global perceptual features, deep features are extracted via a pre-trained convolutional neural network model to represent the high-level semantics. Finally, considering the diversity of the extracted features, an ensemble learning strategy is adopted to promote the generalization performance and prediction accuracy of the proposed model. Experimental results show that, compared with the conventional 2D and SPI quality measurement methods, the proposed method can measure the stitched-specific distortions more accurately, and is more coincident with subjective ratings.

Depth-Image-Based-Rendering (DIBR) procedure aims to create virtual viewpoint using the adjacent viewpoints and the corresponding depth maps for free-viewpoint video applications. However, this processing inevitably introduces visual artifacts which differ from those of natural images, mainly including holes, stretching, and blurring. To accurately evaluate the quality of 3D-synthesized images, a robust blind metric is developed by analysis of Local and Global Statistical Properties (LGSP). On the one hand, for local features, since the DIBR processing tends to damage the amount of information at the disoccluded regions in 3D-synthesized images, we extract the local two-dimensional entropy features from the spatial domain and log-Gabor domain guided by the corresponding saliency map to measure local information variation. Moreover, local statistical features are extracted from fractal dimension maps to capture local texture variations due to significant local texture degradation engendered by the DIBR procedure. On the other hand, for global features, to measure the DIBR-induced geometric distortion, steerable pyramid decomposition is utilized to extract its cross-scale and cross-orientation joint statistical features. Meanwhile, log-energies features are extracted from multi-scale and multi-orientation wavelet subbands to characterize its global blurring. Finally, the combined local and global quality-aware features are utilized to train the quality prediction model using regression function to establish the mapping between visual features and subjective scores. Extensive experiments over three public 3D-synthesized image benchmark databases show that our metric outpaces the newly-developed 3D-synthesized image metrics and the popular 2D image metrics.