Visual quality and algorithm efficiency are two main interests in video frame interpolation. We propose a hybrid task-based convolutional neural network for fast and accurate frame interpolation of 4K videos. The proposed method synthesizes low-resolution frames, then reconstructs high-resolution frames in a coarse-to-fine fashion. We also propose edge loss, to preserve high-frequency information and make the synthesized frames look sharper. Experimental results show that the proposed method achieves state-of-the-art performance and performs 2.69x faster than the existing methods that are operable for 4K videos, while maintaining comparable visual and quantitative quality.
Recently, video frame interpolation research developed with a convolutional neural network has shown remarkable results. However, these methods demand huge amounts of memory and run time for high-resolution videos, and are unable to process a 4K frame in a single pass. In this paper, we propose a fast 4K video frame interpolation method, based upon a multi-scale optical flow reconstruction scheme. The proposed method predicts low resolution bi-directional optical flow, and reconstructs it into high resolution. We also proposed consistency and multi-scale smoothness loss to enhance the quality of the predicted optical flow. Furthermore, we use adversarial loss to make the interpolated frame more seamless and natural. We demonstrated that the proposed method outperforms the existing state-of-the-art methods in quantitative evaluation, while it runs up to 4.39× faster than those methods for 4K videos.
Convolutional neural network-based single image super-resolution (SISR) involves numerous parameters and high computational expenses to ensure improved performance, limiting its applicability in resource-constrained devices such as mobile phones. Knowledge distillation (KD), which transfers useful knowledge from a teacher network to a student network, has been investigated as a method to make networks more efficient in terms of performance. To this end, feature distillation (FD) has been utilized in KD to minimize the Euclidean distance-based loss of feature maps between teacher and student networks. However, this technique does not adequately consider the effective and meaningful delivery of knowledge from the teacher to the student network to improve the latter’s performance under given network capacity constraints. In this study, we propose a feature-domain adaptive contrastive distillation (FACD) method to train lightweight student SISR networks efficiently. We highlight the limitations of existing FD methods in terms of Euclidean distance-based loss, and propose a feature-domain contrastive loss, which causes student networks to learn richer information from the teacher’s representation in the feature domain. We also implement adaptive distillation that performs distillation selectively depending on the conditions of the training patches. Experimental results showed that the proposed FACD scheme improves student enhanced deep residual networks and residual channel attention networks not only in terms of the peak signal-to-noise ratio on all benchmark datasets and scales but also in terms of subjective image quality, compared to the conventional FD approaches. Code will be release at https://github.com/hcmoon0613/FACD.
The You Only Look Once (YOLO) series stands out for its exceptional scalability, enabling seamless deployment on a variety of diverse software and hardware platforms. This scalability has driven its utilization in numerous industrial sites. Recently, there has been an increasing focus on developing quantization-friendly architectures, especially for INT8 inference, to support real-time processing on low-power devices such as mobile platforms. In this paper, we propose the simple and novel approach to enhance the performance of the YOLOv6 model, a widely used object detector in industrial applications, by incorporating skip connections in selected re-parameterization blocks to achieve a quantization-friendly architecture. In addition, we introduce a regression normalization method to address the performance degradation in the head part that often occurs during the TFLite INT8 conversion for mobile environments. The proposed YOLOv6+ architecture outperforms the original YOLOv6 and its successor YOLOv8 by achieving comparable speed in FP/INT8 precision inference while improving mAP performance and enhancing quantization-friendliness. Furthermore, the regression normalization method effectively mitigates performance degradation during TFLite INT8 conversion and is verified to be applicable to other recently developed YOLO series models.
This paper presents the advanced motion-constrained AV1 encoder for a 360 8K VR streaming application. Currently, 360 8K VR streaming faces problems in limited network bandwidth, limited server-side computing power that makes difficult to serve ultra-high resolution videos to a large number of clients, and limiting power of a consumer electronic device to decode the stream. We propose a motion-constrained AV1 encoder that can solve these problems. The proposed method is composed of three different restrictions: motion vector search range, AV1 encoding options, and constrained loop filters in the boundary region of tiles. The experimental results show the average BDBR overhead of the proposed method with various tiles is 10.00% with saving 17.30% encoding time compared to the existing AV1 encoder. Also, the proposed AV1 encoder improves the BDBR by an average of 12.50% compared to a motion-constrained HEVC encoder.