Video coding for machines is an emerging area within video compression technology that has recently attracted considerable research attention. Within the ISO/IEC standardization activities, efforts are underway to develop a new standard optimized for machine vision tasks, rather than just for human-oriented video consumption. In this context, the novel contribution of this work is the application of neural networks to perform chroma synthesis at the decoder side, thus eliminating the need for direct chroma transmission. This concept has been implemented and validated in the Video Coding for Machines Reference Software (VCM-RS), a test model developed by the MPEG Video group, which is briefly described for completeness. Experimental results reported in this paper show that our approach significantly reduces the number of bits required for video representation, achieving bitrate savings of up to 60% for certain video sequences. The presented proposal has been accepted in the current version of the specification of the upcoming VCM standard.

The application of machine learning to video coding is generally studied in two main approaches: end-to-end video coding using deep neural networks and classic hybrid codecs with individual tools implemented using such networks. This work exploits the latter approach, where a trained Artificial Neural Network (ANN) is used for fast implementation of the search for the partitioning of Coding Tree Units (CTU) into Coding Units (CUs) and Prediction Units (PUs). The proposed approach differs from the previous ones, among other factors, by the application of an ANN with probabilistic soft outputs, which allows for assessing the probability of particular division patterns. This is followed by a decision algorithm that selects more than one candidate division pattern from the most probable patterns provided by ANN. Two variants of the ANN model are considered, out of which the extended one involves deep partitioning at the PU-level. The number of selected division pattern candidates is controlled by a single parameter – uncertainty range – that allows for choosing a trade-off between the complexity and the loss of coding efficiency. This feature is particularly important for broadcasting applications, where processing must be adjusted to the current computational load and resource availability. The experiments demonstrate that the proposed approach yields good results using much smaller ANNs than those described in the references. This makes the proposed approach well-suited for the usage on edge servers where GPU may not be available. The source code for the respective HEVC codec software implementation is provided.

In the intra-frame video coding, an image is divided into small blocks, and the actual coding is performed individually in these blocks. In this paper, the process is considered in the context of the widely used HEVC compression, where the optimum choice of the division is crucial for the ratedistortion performance. Unfortunately, the search for such optimum division needs very many operations, and is done on the basis of “try and check” approach in the classic implementations. The idea of the paper is to replace this complex part of the encoder by a neural network, and some variants of the potential neural networks are studied and compared in the paper. For the chosen network, the complexity of the encoder is vastly reduced at the cost of negligible loss in the rate-distortion performance. These features are demonstrated using an extensive set of frames from many test video sequences.

In HEVC compression is performed in Coding Units (CUs) being pixel blocks of a size adaptively chosen according to the local content within a video frame. Nearoptimum selection of the frame partition into CUs is crucial for the coding efficiency. A huge number of partitioning schemes is available and the optimum partitioning scheme is obtained in an iterative computation-heavy procedure in a classic HEVC encoder. In order to reduce the encoding time and the encoding energy, a few approaches have been proposed with the use of neural networks (NNs). These approaches demonstrate a significant reduction of the encoding time and a negligible increase of the bitrate as compared to the traditional iterative approach. Nevertheless, they use very large neural networks whereas it is demonstrated in this paper that much smaller neural networks provide similar results encoding tome reduction with the similar bitrate reduction.

Due to the growing complexity of video encoders, the optimization of the parameters of the encoding process is becoming an important issue. In recent years, this has become an important field of application of neural networks. Artificial neural networks in video encoders are used to accelerate the video encoder operation. This paper demonstrates the use of different ResNet- and DenseNet-type architectures to accelerate the CTU partitioning algorithm in HEVC in All Intra mode. The paper demonstrates the results of an exhaustive evaluation of different proposed architectures, considering compression efficiency, network size, and encoding time reduction. Multiple pros and cons of the proposed architectures are presented in the Conclusions, considering various limitations that may be important for a given application, like hardware-constrained sensor networks or standalone small devices operating with images and videos.