With the advancement of sensors, image and video processing have developed for use in the visual sensing area. Among them, video super-resolution (VSR) aims to reconstruct high-resolution sequences from low-resolution sequences. To use consecutive contexts within a low-resolution sequence, VSR learns the spatial and temporal characteristics of multiple frames of the low-resolution sequence. As one of the convolutional neural network-based VSR methods, we propose a deformable convolution-based alignment network (DCAN) to generate scaled high-resolution sequences with quadruple the size of the low-resolution sequences. The proposed method consists of a feature extraction block, two different alignment blocks that use deformable convolution, and an up-sampling block. Experimental results show that the proposed DCAN achieved better performances in both the peak signal-to-noise ratio and structural similarity index measure than the compared methods. The proposed DCAN significantly reduces the network complexities, such as the number of network parameters, the total memory, and the inference speed, compared with the latest method.

Versatile Video Coding (VVC), the state-of-the-art video coding standard, was developed by the Joint Video Experts Team (JVET) of ISO/IEC Moving Picture Experts Group (MPEG) and ITU-T Video Coding Experts Group (VCEG) in 2020. Although VVC can provide powerful coding performance, it requires tremendous computational complexity to determine the optimal mode decision during the encoding process. In particular, VVC adopted the bi-prediction with CU-level weight (BCW) as one of the new tools, which enhanced the coding efficiency of conventional bi-prediction by assigning different weights to the two prediction blocks in the process of inter prediction. In this study, we investigate the statistical characteristics of input features that exhibit a correlation with the BCW and define four useful types of categories to facilitate the inter prediction of VVC. With the investigated input features, a lightweight neural network with multilayer perceptron (MLP) architecture is designed to provide high accuracy and low complexity. We propose a fast BCW mode decision method with a lightweight MLP to reduce the computational complexity of the weighted multiple bi-prediction in the VVC encoder. The experimental results show that the proposed method significantly reduced the BCW encoding complexity by up to 33% with unnoticeable coding loss, compared to the VVC test model (VTM) under the random-access (RA) configuration.

The increasing demand for high-quality audio content in applications such as media streaming and immersive audio services has driven the development of efficient audio coding techniques. Among these techniques, entropy coding is an essential component for the lossless compression of quantized spectral coefficients, which constitute the largest portion of the compressed audio bitstream. As the latest audio coding standard, Unified Speech and Audio Coding (USAC) has adopted ContextAdaptive Arithmetic Coding (CAAC) to employ its entropy coding method. Although CAAC can provide efficient lossless coding, it has several limitations, including bit overhead caused by variations in bit lengths between coefficients and considerable memory requirements for storing multiple context tables. To reduce excessive memory consumption, this study proposes Dynamic Bit-plane Arithmetic Coding (DBAC) as a new entropy coding method for quantized spectral coefficients. DBAC mitigates the bit overhead by using escape symbols for binarized coefficients and reduces memory usage while maintaining coding performance, even with marginal coding gains. In addition, the generation and selection methods of newly designed multiple contexts are also introduced to optimize arithmetic coding on the proposed dynamic bit-plane of DBAC. Experimental results demonstrate that the proposed method achieves 30% memory reduction with comparable coding performance compared to USAC.

At the ImageNet Large Scale Visual Re-Conversion Challenge (ILSVRC), a 2012 global image recognition contest, the University of Toronto Supervision team led by Prof

Video super-resolution (VSR) aims to reconstruct high-resolution (HR) videos from low-resolution (LR) inputs by utilizing spatio-temporal correlations across consecutive LR frames. While recent advances in deep learning, particularly transformer-based architecture, have substantially improved VSR performance, maintaining spatio-temporal coherence remains a critical challenge. To address this issue, we propose a novel U-Net based spatial transformer module (USTM) that can be seamlessly integrated into the reconstruction stage of existing VSR frameworks. The proposed USTM combines a spatial transformer with a U-Net structure to extract fine-grained spatio-temporal features to enhance the reconstruction of complex motions and texture patterns. Extensive ablation studies were conducted to identify the optimal configuration and verify the contribution of each USTM component. For performance evaluations, USTM was incorporated into multiple representative VSR methods. Experimental results demonstrate that integrating USTM consistently enhances PSNR and SSIM scores on benchmark datasets, including REDS, Vimeo-90K, and Vid4. Furthermore, visual comparisons highlight the superiority of the proposed method, particularly in high-frequency regions, compared to baseline VSR methods without USTM.

Recently, the demand for immersive videos has surged with the expansion of virtual reality, augmented reality, and metaverse technologies. As an international standard, moving picture experts group (MPEG) has developed MPEG immersive video (MIV) to efficiently transmit large-volume immersive videos. The MIV encoder generates atlas videos to convert extensive multi-view videos into low-bitrate formats. When these atlas videos are compressed using conventional video codecs, compression artifacts often appear in the reconstructed atlas videos. To address this issue, this study proposes a feature-extraction-based convolutional neural network (FECNN) to reduce the compression artifacts during MIV atlas video transmission. The proposed FECNN uses quantization parameter (QP) maps and depth information as inputs and consists of shallow feature extraction (SFE) blocks and deep feature extraction (DFE) blocks to utilize layered feature characteristics. Compared to the existing MIV, the proposed method improves the Bjontegaard delta bit-rate (BDBR) by −4.12% and −6.96% in the basic and additional views, respectively.