Wenbo Bao

Wenbo Bao

Auto-generated profile, based on publicly available information

Wenbo Bao

Department of Electronic Engineering, Institute of Image Communication and Network Engineering, Shanghai, Shanghai China 200240

Research Focus & Metrics

Academic Career Length

9Years (2018-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
17 September 2019 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Verified
Journal Article
Closed access
WBWenbo Bao
Wenbo Bao
WLWei-Sheng Lai
Wei-Sheng Lai
XZXiaoyun Zhang
Xiaoyun Zhang
ZGZhiyong Gao
Zhiyong Gao
MYMing Yang
Ming Yang
Motion estimation (ME) and motion compensation (MC) have been widely used for classical video frame interpolation systems over the past decades. Recently, a number of data-driven frame interpolation methods based on convolutional neural networks have been proposed. However, existing learning based methods typically estimate either flow or compensation kernels, thereby limiting performance on both computational efficiency and interpolation accuracy. In this work, we propose a motion estimation and compensation driven neural network for video frame interpolation. A novel adaptive warping layer is developed to integrate both optical flow and interpolation kernels to synthesize target frame pixels. This layer is fully differentiable such that both the flow and kernel estimation networks can be optimized jointly. The proposed model benefits from the advantages of motion estimation and compensation methods without using hand-crafted features. Compared to existing methods, our approach is computationally efficient and able to generate more visually appealing results. Furthermore, the proposed MEMC-Net can be seamlessly adapted to several video enhancement tasks, e.g., super-resolution, denoising, and deblocking. Extensive quantitative and qualitative evaluations demonstrate that the proposed method performs favorably against the state-of-the-art video frame interpolation and enhancement algorithms on a wide range of datasets.

9 April 2018 in IEEE Transactions on Image Processing
Verified
Journal Article
Closed access
WBWenbo Bao
Wenbo Bao
XZXiaoyun Zhang
Xiaoyun Zhang
LCLi Chen
Li Chen
LDLianghui Ding
Lianghui Ding
ZGZhiyong Gao
Zhiyong Gao
This paper proposes a novel frame rate up-conversion method through high-order model and dynamic filtering (HOMDF) for video pixels. Unlike the constant brightness and linear motion assumptions in traditional methods, the intensity and position of the video pixels are both modeled with high-order polynomials in terms of time. Then, the key problem of our method is to estimate the polynomial coefficients that represent the pixel's intensity variation, velocity, and acceleration. We propose to solve it with two energy objectives: one minimizes the auto-regressive prediction error of intensity variation by its past samples, and the other minimizes video frame's reconstruction error along the motion trajectory. To efficiently address the optimization problem for these coefficients, we propose the dynamic filtering solution inspired by video's temporal coherence. The optimal estimation of these coefficients is reformulated into a dynamic fusion of the prior estimate from pixel's temporal predecessor and the maximum likelihood estimate from current new observation. Finally, frame rate up-conversion is implemented using motion-compensated interpolation by pixel-wise intensity variation and motion trajectory. Benefited from the advanced model and dynamic filtering, the interpolated frame has much better visual quality. Extensive experiments on the natural and synthesized videos demonstrate the superiority of HOMDF over the state-of-the-art methods in both subjective and objective comparisons.

16 November 2020 in IEEE Transactions on Image Processing
Verified
Journal Article
Closed access
WSWang Shen
Wang Shen
WBWenbo Bao
Wenbo Bao
GZGuangtao Zhai
Guangtao Zhai
LCLi Chen
Li Chen
XMXiongkuo Min
Xiongkuo Min
ZGZhiyong Gao
Zhiyong Gao
Video frame interpolation aims to improve users’ watching experiences by generating high-frame-rate videos from low-frame-rate ones. Existing approaches typically focus on synthesizing intermediate frames using high-quality reference images. However, the captured reference frames may suffer from inevitable spatial degradations such as motion blur, sensor noise, etc. Few studies have approached the joint video enhancement problem, namely synthesizing high-frame-rate and high-quality results from low-frame-rate degraded inputs. In this paper, we propose a unified optimization framework for video frame interpolation with spatial degradations. Specifically, we develop a frame interpolation module with a pyramid structure to cyclically synthesize high-quality intermediate frames. The pyramid module features adjustable spatial receptive field and temporal scope, thus contributing to controllable computational complexity and restoration ability. Besides, we propose an inter-pyramid recurrent module to connect sequential models to exploit the temporal relationship. The pyramid module integrates the recurrent module, thus can iteratively synthesize temporally smooth results. And the pyramid modules share weights across iterations, thus it does not expand the model’s parameter size. Our model can be generalized to several applications such as up-converting the frame rate of videos with motion blur, reducing compression artifacts, and jointly super-resolving low-resolution videos. Extensive experimental results demonstrate that our method performs favorably against state-of-the-art methods on various video frame interpolation and enhancement tasks.

1 April 2020 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Verified
Journal Article
Closed access
MCMing Cheng
Ming Cheng
ZMZhan Ma
Zhan Ma
MAM. Salman Asif
M. Salman Asif
YXYiling Xu
Yiling Xu
HLHaojie Liu
Haojie Liu
JSJun Sun
Jun Sun
This paper presents a dual camera system for high spatiotemporal resolution (HSTR) video acquisition, where one camera shoots a video with high spatial resolution and low frame rate (HSR-LFR) and another one captures a low spatial resolution and high frame rate (LSR-HFR) video. Our main goal is to combine videos from LSR-HFR and HSR-LFR cameras to create an HSTR video. We propose an end-to-end learning framework, AWnet, mainly consisting of a FlowNet and a FusionNet that learn an adaptive weighting function in pixel domain to combine inputs in a frame recurrent fashion. To improve the reconstruction quality for cameras used in reality, we also introduce noise regularization under the same framework. Our method has demonstrated noticeable performance gains in terms of both objective PSNR measurement in simulation with different publicly available video and light-field datasets and subjective evaluation with real data captured by dual iPhone 7 and Grasshopper3 cameras. Ablation studies are further conducted to investigate and explore various aspects (such as reference structure, camera parallax, exposure time, etc) of our system to fully understand its capability for potential applications.

7 March 2019 in IEEE Transactions on Image Processing
Verified
Journal Article
Closed access
WBWenbo Bao
Wenbo Bao
XZXiaoyun Zhang
Xiaoyun Zhang
LCLi Chen
Li Chen
ZGZhiyong Gao
Zhiyong Gao
Recent studies on optical flow typically focus on the estimation of the single flow field in-between a pair of images but pay little attention to the multiple consecutive flow fields in a longer video sequence. In this paper, we propose an efficient video optical flow estimation method by exploiting the temporal coherence and context dynamics under a Kalman filtering system. In this system, pixel’s motion flow is first formulated as a second-order time-variant state vector, and then optimally estimated according to the measurement and system noise levels within the system by maximum a posteriori criteria. Specifically, we evaluate the measurement noise according to the flow’s temporal derivative, spatial gradient, and warping error. And we determine the system noise based on the similarity of contextual information, which is represented by the compact features learned by pre-trained convolutional neural networks. The context-aware Kalman filtering helps improve the robustness of our method against abrupt change of light and occlusion/dis-occlusion in complicated scenes. Experimental results and analyses on the MPI Sintel, Monkaa and Driving video datasets demonstrate that the proposed method performs favorably against the state-of-the-art approaches.

14 August 2020 in arXiv
Verified
Preprint
Open access
PDF file
CGCong Geng
Cong Geng
JWJia Wang
Jia Wang
LCLi Chen
Li Chen
WBWenbo Bao
Wenbo Bao
CCChu Chu
Chu Chu
ZGZhiyong Gao
Zhiyong Gao
In this paper, we propose a method to learn a minimizing geodesic within a data manifold. Along the learned geodesic, our method can generate high-quality interpolations between two given data samples. Specifically, we use an autoencoder network to map data samples into latent space and perform interpolation via an interpolation network. We add prior geometric information to regularize our autoencoder for the convexity of representations so that for any given interpolation approach, the generated interpolations remain within the distribution of the data manifold. Before the learning of a geodesic, a proper Riemannianmetric should be defined. Therefore, we induce a Riemannian metric by the canonical metric in the Euclidean space which the data manifold is isometrically immersed in. Based on this defined Riemannian metric, we introduce a constant speed loss and a minimizing geodesic loss to regularize the interpolation network to generate uniform interpolation along the learned geodesic on the manifold. We provide a theoretical analysis of our model and use image translation as an example to demonstrate the effectiveness of our method.