The real-time simulation technology of large-scale open sea surfaces has been of great importance in fields such as computer graphics, ocean engineering, and national security. However, existing technologies typically have performance requirements or platform limitations, and the two are often impossible to balance. Based on the camera-view-based screen space level of detail strategy and virtual camera pose adaptive filtering strategy proposed in this article, we have developed a fast and comprehensive solution for rendering large-scale open sea surfaces. This solution is designed to work without the need for special hardware extensions, making it easy to deploy across various platforms. Additionally, it enhances the degrees of freedom of virtual camera movement. After conducting performance tests under various camera poses, our filtering strategy was found to be effective. Notably, the time cost of simulation using 60 waves at the height of 6 m above sea level was only 0.184 ms. In addition, we conducted comparative experiments with four state-of-the-art algorithms currently in use, and our solution outperformed the others with the best performance and suboptimal visual effects. These results demonstrate the superiority of our approach in terms of both efficiency and effectiveness.
We present MobileSky , the first automatic method for real-time high-quality sky replacement for mobile AR applications. The primary challenge of this task is how to extract sky regions in camera feed both quickly and accurately. While the problem of sky replacement is not new, previous methods mainly concern extraction quality rather than efficiency, limiting their application to our task. We aim to provide higher quality, both spatially and temporally consistent sky mask maps for all camera frames in real time. To this end, we develop a novel framework that combines a new deep semantic network called FSNet with novel post-processing refinement steps. By leveraging IMU data, we also propose new sky-aware constraints such as temporal consistency, position consistency, and color consistency to help refine the weakly classified part of the segmentation output. Experiments show that our method achieves an average of around 30 FPS on off-the-shelf smartphones and outperforms the state-of-the-art sky replacement methods in terms of execution speed and quality. In the meantime, our mask maps appear to be visually more stable across frames. Our fast sky replacement method enables several applications, such as AR advertising, art making, generating fantasy celestial objects, visually learning about weather phenomena, and advanced video-based visual effects. To facilitate future research, we also create a new video dataset containing annotated sky regions with IMU data.
8 July 2024 in International Journal of Digital Earth