7 October 2020 in IEEE Transactions on Computational Imaging
Ground based long-range passive imaging systems often suffer from degraded image quality due to a turbulent atmosphere. While methods exist for removing such turbulent distortions, many are limited to static sequences which cannot be extended to dynamic scenes. In addition, the physics of the turbulence is often not integrated into the image reconstruction algorithm. In this paper, we present a unified method for atmospheric turbulence mitigation in both static and dynamic sequences. We are able to achieve better results compared to existing methods by utilizing (i) a novel space-time non-local averaging method to construct a reliable reference frame, (ii) a geometric consistency and a sharpness metric to generate the lucky frame, (iii) a physics-constrained prior model of the point spread function for blind deconvolution. Experimental results based on synthetic and real long-range turbulence sequences validate the performance of the proposed method.
16 January 2024 in IEEE Transactions on Computational Imaging
Restoring images distorted by atmospheric turbulence is a ubiquitous problem in long-range imaging applications. While existing deep-learning-based methods have demonstrated promising results in specific testing conditions, they suffer from three limitations: (1) lack of generalization capability from synthetic training data to real turbulence data; (2) failure to scale, hence causing memory and speed challenges when extending the idea to a large number of frames; (3) lack of a fast and accurate simulator to generate data for training neural networks. In this paper, we introduce the turbulence mitigation transformer (TMT) that explicitly addresses these issues. TMT brings three contributions: Firstly, TMT explicitly uses turbulence physics by decoupling the turbulence degradation and introducing a multi-scale loss for removing distortion, thus improving effectiveness. Secondly, TMT presents a new attention module along the temporal axis to extract extra features efficiently, thus improving memory and speed. Thirdly, TMT introduces a new simulator based on the Fourier sampler, temporal correlation, and flexible kernel size, thus improving our capability to synthesize better training data. TMT outperforms state-of-the-art video restoration models, especially in generalizing from synthetic to real turbulence data.
1 August 2020 in Optical Engineering
<!-- *** Custom HTML *** --> Simulating atmospheric turbulence is an essential task for evaluating turbulence mitigation algorithms and training learning-based methods. Advanced numerical simulators for atmospheric turbulence are available, but they require evaluating wave propagation, which is computationally expensive. We present a propagation-free method for simulating imaging through turbulence with applications in ground-to-ground imaging. The key idea behind our work is a method to draw intermodal and spatially correlated Zernike coefficients. By establishing the equivalence of the angle-of-arrival correlation by Basu, McCrae, and Fiorino (2015) with the multiaperture correlation by Chanan (1992), we show that the Zernike coefficients can be drawn according to a covariance matrix defining the correlations. We propose fast and scalable sampling strategies to draw these samples. The method allows us to compress the wave propagation problem into a sampling problem, hence making the simulator significantly faster than existing ones. Experimental results show that the simulator has an excellent match with the theory and real turbulence data. We anticipate the simulator, which balances speed and accuracy, will be a useful tool for various computational imaging applications.
2 December 2022 in IEEE Transactions on Computational Imaging
Numerical simulation of atmospheric turbulence is one of the biggest bottlenecks in developing computational techniques for solving the inverse problem in long-range imaging. The classical split-step method is based upon numerical wave propagation which splits the propagation path into many segments and propagates every pixel in each segment individually via the Fresnel integral. This repeated evaluation becomes increasingly time-consuming for larger images. As a result, the split-step simulation is often done only on a sparse grid of points followed by an interpolation to the other pixels. Even so, the computation is expensive for real-time applications. In this paper, we present a new simulation method that enables real-time processing over a dense grid of points. Building upon the recently developed multi-aperture model and the phase-to-space transform, we overcome the memory bottleneck in drawing random samples from the Zernike correlation tensor. We show that the cross-correlation of the Zernike modes has an insignificant contribution to the statistics of the random samples. By approximating these cross-correlation blocks in the Zernike tensor, we restore the homogeneity of the tensor which then enables Fourier-based random sampling. On a $512\times 512$ image, the new simulator achieves 0.025 seconds per frame over a dense field. On a $3840 \times 2160$ image which would have taken 13 hours to simulate using the split-step method, the new simulator can run at approximately 60 seconds per frame.
30 October 2023 in Foundations and Trends in Computer Graphics and Vision
Since the seminal work of Andrey Kolmogorov in the early 1940’s, imaging through atmospheric turbulence has grown from a pure scientific pursuit to an important subject across a multitude of civilian, space-mission, and national security applications. Fueled by the recent advancement of deep learning, the field is further experiencing a new wave of momentum of applying these learning-based techniques to the problem. However, because of the complexity of the physics of atmospheric turbulence, significant gaps remain to be filled before the power of deep learning can be fully unleashed. In particular, the goal of building the most accurate turbulence model to mimic nature is gradually shifted to designing a compromised model that can maximize the image reconstruction performance. This leads to a new field which this book is trying to explain, Computational Imaging Through Atmospheric Turbulence. The goal of this book is to present the basic concepts of turbulence physics while framing it under the theme of computational imaging. Emphasis is put on elaborating the principles of how waves propagate through atmospheric turbulence and propagation-free approaches to reproduce the effect without needing wave propagation equations. This allows for a much faster simulation while preserving the physics of turbulence, hence creating the possibility of integrating turbulence physics into the design of image reconstruction algorithms. The book is written for readers with an image processing background who are seeking to understand the physics of turbulence. Connections with deep learning are emphasized throughout the book.
11 March 2024 in IEEE Transactions on Signal Processing
A spatially varying blur kernel h (x, u) is specified by an input coordinate u ∈ $\mathbb{R}^{2}$ and an output coordinate x ∈ $\mathbb{R}^{2}$ . For computational efficiency, we sometimes write h (x, u) as a linear combination of spatially invariant basis functions. The associated pixelwise coefficients, however, can be indexed by either the input coordinate or the output coordinate. While appearing subtle, the two indexing schemes will lead to two different forms of convolutions known as scattering and gathering , respectively. We discuss the origin of the operations. We discuss conditions under which the two operations are identical. We show that scattering is more suitable for simulating how light propagates and gathering is more suitable for image filtering such as denoising.