8 April 2024 in IEEE Transactions on Circuits and Systems for Video Technology
Novel-view synthesis with sparse input views is important for practical applications such as AR/VR and autonomous driving. Many works in this field have already integrated depth information into NeRF, utilizing depth priors for assistance in geometric and spatial understanding. However, most existing work tends to either overlook the inaccuracies in depth maps or only handle them roughly, limiting the effectiveness of the synthesis. To address this issue, we propose a depth-guided robust point cloud fusion NeRF for sparse input synthesis. We first construct a point cloud for each input view, with a novel point cloud representation based on learnable matrices and vectors. Then, through an additional lightweight scene fusion network, we fuse the point clouds from each input view to build a point cloud of the entire scene. By optimizing the point cloud representation and scene fusion network, inaccuracies in the depth map can be adjusted and refined, thereby achieving a more precise perception of the overall scene. Each voxel in the scene is determined by referencing the fused point cloud to establish its density and appearance. Experimental results demonstrate that our method outperforms state-of-the-art baselines.
18 April 2024 in Displays
13 January 2022 in IEEE Transactions on Multimedia
In this paper, we propose a deep learning based sensor-driven method for online video stabilization. This method utilizes euler angle and acceleration estimated from the gyroscope and accelerator to assist stable video reconstruction. We introduce two simple sub-networks for trajectory optimization, the first network exploits real unstable trajectory and acceleration of camera to detect shooting scenarios. This network also generates an attention mask to choose scenario-specific features adaptively. Then the second network predicts smooth camera paths based on real unstable trajectory using LSTM under the supervision of the former mask. The output of trajectory optimization network is filtered with a two-step modification process to guarantee the smoothness. The real and smoothed camera paths are then utilized as guidance to projectively generate stable frames. We also capture videos with sensor data covering seven typical shooting scenarios, and design a groundtruth generation method to construct pseud-labels. Moreover, the trajectory smoothing network allows 3 or 10 frames buffer as the future information to construct a lookahead filter. Experimental results show that our online method could outperform other offline state-of-the-art methods in several shaky video clips with fewer buffer frames over both general and low-quality videos. Furthermore, our method could effectively reduce running times without image content analysis, and the stabilization efficiency reaches 25 fps on 1080p videos.