Rate control plays an important role in video coding systems, which makes the output bitrate of a video encoder equal to the target bitrate while minimizing the distortion of the compressed video. However, most of the existing rate control schemes achieve accurate bitrate control at the loss of rate-distortion (R-D) performance. This paper proposes an effective frame level bit allocation method to improve the R-D performance whilst maintaining the high accuracy of the bitrate control in HEVC. First, an improved R-D model is presented at the frame level, which, by making use of the information of encoded frames more completely, achieves higher accuracy with lower bitrate mismatch rate. Second, a group of picture (GOP) level bit allocation approach is introduced with the consideration of the temporal R-D dependency among different GOPs, which can further enhance the R-D performance. Finally, to achieve optimal R-D performance globally, a formulation for optimal frame level bit allocation is developed with a GOP level Lagrange multiplier introduced, which takes into account the coding effects of a current frame on the other frames within the same GOP. A scheme of recursive Taylor expansion is employed to find the GOP level Lagrange multiplier. Experimental results demonstrate that the proposed method can achieve considerable R-D performance improvement and accurate bitrate control. Specifically, our method shows 3.9% and 3.8% BD-rate saving in average compared against the HEVC reference software HM16.7 with rate control in the low-delay B and low-delay P coding structures, respectively.

Pose based hand gesture recognition has been widely studied in the recent years. Compared with full body action recognition, hand gesture involves joints that are more spatially closely distributed with stronger collaboration. This nature requires a different approach from action recognition to capturing the complex spatial features. Many gesture categories, such as “Grab” and “Pinch”, have very similar motion or temporal patterns posing a challenge on temporal processing. To address these challenges, this paper proposes a two-stream neural network with one stream being a self-attention based graph convolutional network extracting the short-term temporal information and hierarchical spatial information, and the other being a residual-connection enhanced bidirectional Independently Recurrent Neural Network for extracting long-term temporal information. The self-attention based graph convolutional network has a dynamic self-attention mechanism to adaptively exploit the relationships of all hand joints in addition to the fixed topology and local feature extraction in the graph convolutional network. The proposed method effectively takes advantage of the graph convolutional network and independently recurrent neural network to capture the temporal-spatial information. The widely used Dynamic Hand Gesture dataset (two evaluation protocols) and First-Person Hand Action dataset are used to validate its effectiveness and our method achieves state-of-the-art performance with 96.31%, 94.05% and 90.26%, respectively, in terms of recognition accuracy.

Due to the widely used inter prediction in the current video coding standards, encoding of units among different frames are temporally dependent on each other, which implies that rate-distortion optimization (RDO) of one unit may affect the optimization of the following units in the temporal domain. To achieve optimal coding solution for a given video sequence, temporal dependency among units needs to be considered in the RDO process, which is known as the temporally dependent RDO problem (TD-RDO). The hierarchical coding structure (HCS) employed in the High Efficiency Video Coding (HEVC) standard further complicates this problem by grouping frames into different layers with different coding strategies, leading to a complex temporal relationship. In our earlier work, we addressed TD-RDO for the low delay HCS (LD-HCS), where only uni-prediction is considered. This work aims to deal with more complicated TD-RDO under random access HCS (RA-HCS), where both uni-prediction and bi-prediction are considered, making the temporal relationship even more intricate. In this paper, the temporal dependency introduced in the RA-HCS is thoroughly examined and an RA-based TD-RDO scheme is formulated for each layer by modeling distortion temporal propagation under different prediction types. The representation of the global Lagrange multiplier, which is hard to be determined under LD-HCS, can be obtained based on the formulation of the highest layer under RA-HCS. Furthermore, the effect of random access point pictures is considered in the RA-based TD-RDO scheme. The proposed method can be realized by adapting the Lagrange multiplier only or combined with adjusting quantization parameter (QP) for better results in terms of BD-rate saving. Experimental results show that under RA-HCS, the proposed method, by adapting the Lagrange multiplier, can achieve about 2.2% bitrate savings in average. With multi-QP optimization, an average BD-rate gain of 5.2% can be obtained.

Smartphone-sensors-based human activity recognition is attracting increasing interest due to the popularization of smartphones. It is a difficult long-range temporal recognition problem, especially with large intraclass distances such as carrying smartphones at different locations and small interclass distances such as taking a train or subway. To address this problem, we propose a new framework of combining short-term spatial/frequency feature extraction and a long-term independently recurrent neural network (IndRNN) for activity recognition. Considering the periodic characteristics of the sensor data, short-term temporal features are first extracted in the spatial and frequency domains. Then, the IndRNN, which can capture long-term patterns, is used to further obtain the long-term features for classification. Given the large differences when the smartphone is carried at different locations, a group-based location recognition is first developed to pinpoint the location of the smartphone. The Sussex-Huawei Locomotion (SHL) dataset from the SHL Challenge is used for evaluation. An earlier version of the proposed method won the second place award in the SHL Challenge 2020 (first place if not considering the multiple models fusion approach). The proposed method is further improved in this paper and achieves 80.72% accuracy, better than the existing methods using a single model.

In a composite image, the foreground and background are filmed under different scenarios, such as different lighting conditions, causing inconsistency and reducing the overall realism of the image. Image harmonization aims to generate visually realistic composite images by adjusting the foreground to the background conditions while maintaining the structure. Existing methods focus on adjusting the foreground object by directly training the foreground generation network with the ground truth, neglecting the different roles of the illumination and structure of the foreground in image harmonization. Moreover, the use of background, except for providing illumination, is not thoroughly investigated in this task. In this paper, we propose a structure-preserving and illumination-consistent cycle (SP-IC cycle) framework for image harmonization by exploring the illumination and structure of both the foreground and background. It achieves image harmonization by specifically changing the illumination and keeping the structure instead of ambiguously changing the foreground. Then, an illumination-consistent foreground harmonization cycle is developed to change the foreground illumination, while a structure-preserving cycle is designed to keep the foreground structure. Background information is explored in both cycles to assist in decomposing the illumination and structure of the foreground. In addition, the proposed SP-IC cycle framework can be applied to any image harmonization method to further boost its performance. Experimental results demonstrate that our method achieves better harmonious image quality than state-of-the-art methods, especially on an illumination-varying dataset.

In-loop filtering is used in video coding to process the reconstructed frame in order to remove blocking artifacts. With the development of convolutional neural networks (CNNs), CNNs have been explored for in-loop filtering considering it can be treated as an image de-noising task. However, in addition to being a distorted image, the reconstructed frame is also obtained by a fixed line of block based encoding operations in video coding. It carries coding-unit based coding distortion of some similar characteristics. Therefore, in this paper, we address the filtering problem from two aspects, (i) global appearance restoration for disrupted texture and (ii) local coding distortion restoration caused by fixed pipeline of coding. Accordingly, a three-stream global appearance and local coding distortion based fusion network is developed with a high-level global feature stream, a high-level local feature stream and a low-level local feature stream. Ablation study is conducted to validate the necessity of different features, demonstrating that the global features and local features can complement each other in filtering and achieve better performance when combined. To the best of our knowledge, we are the first one that clearly characterizes the video filtering process from the above global appearance and local coding distortion restoration aspects with experimental verification, providing a clear pathway to developing filter techniques. Experimental results demonstrate that the proposed method significantly outperforms the existing single-frame based methods and achieves 13.5%, 11.3%, 11.7% BD-Rate saving on average for AI, LDP and RA configurations, respectively, compared with the HEVC reference software.