1 September 2017 in IEEE Transactions on Image Processing
In this paper, we propose a novel deep convolutional neural network (CNN)-based algorithm for solving ill-posed inverse problems. Regularized iterative algorithms have emerged as the standard approach to ill-posed inverse problems in the past few decades. These methods produce excellent results, but can be challenging to deploy in practice due to factors including the high computational cost of the forward and adjoint operators and the difficulty of hyperparameter selection. The starting point of this paper is the observation that unrolled iterative methods have the form of a CNN (filtering followed by pointwise nonlinearity) when the normal operator (H*H, where H* is the adjoint of the forward imaging operator, H) of the forward model is a convolution. Based on this observation, we propose using direct inversion followed by a CNN to solve normal-convolutional inverse problems. The direct inversion encapsulates the physical model of the system, but leads to artifacts when the problem is ill posed; the CNN combines multiresolution decomposition and residual learning in order to learn to remove these artifacts while preserving image structure. We demonstrate the performance of the proposed network in sparse-view reconstruction (down to 50 views) on parallel beam X-ray computed tomography in synthetic phantoms as well as in real experimental sinograms. The proposed network outperforms total variation-regularized iterative reconstruction for the more realistic phantoms and requires less than a second to reconstruct a 512 × 512 image on the GPU.
9 November 2017 in IEEE Signal Processing Magazine
In this article, we review recent uses of convolutional neural networks (CNNs) to solve inverse problems in imaging. It has recently become feasible to train deep CNNs on large databases of images, and they have shown outstanding performance on object classification and segmentation tasks. Motivated by these successes, researchers have begun to apply CNNs to the resolution of inverse problems such as denoising, deconvolution, superresolution, and medical image reconstruction, and they have started to report improvements over state-of-the-art methods, including sparsity-based techniques such as compressed sensing.
3 May 2018 in IEEE Transactions on Medical Imaging
We present a new image reconstruction method that replaces the projector in a projected gradient descent (PGD) with a convolutional neural network (CNN). Recently, CNNs trained as image-to-image regressors have successfully been used to solve inverse problems in imaging. However, unlike existing iterative image reconstruction algorithms, these CNN-based approaches usually lack a feedback mechanism to enforce that the reconstructed image is consistent with the measurements. We propose a relaxed version of PGD wherein gradient descent enforces measurement consistency, while a CNN recursively projects the solution closer to the space of desired reconstruction images. We show that this algorithm is guaranteed to converge and, under certain conditions, converges to a local minimum of a non-convex inverse problem. Finally, we propose a simple scheme to train the CNN to act like a projector. Our experiments on sparseview computed-tomography reconstruction show an improvement over total variation-based regularization, dictionary learning, and a state-of-the-art deep learning-based direct reconstruction technique.
13 July 2021 in IEEE Transactions on Computational Imaging
We present CryoGAN, a new paradigm for single-particle cryo-electron microscopy (cryo-EM) reconstruction based on unsupervised deep adversarial learning. In single-particle cryo-EM, the structure of a biomolecule needs to be reconstructed from a large set of noisy tomographic projections with unknown orientations. Current reconstruction techniques are based on a marginalized maximum-likelihood formulation that requires calculations over the set of all possible poses for each projection image, a computationally demanding procedure. Our approach is to seek a 3D structure that has simulated projections that match the real data in a distributional sense, thereby sidestepping pose estimation or marginalization. We prove that, in an idealized mathematical model of cryo-EM, this approach results in recovery of the correct structure. Motivated by distribution matching, we propose CryoGAN, a specialized GAN that consists of a 3D structure, a cryo-EM physics simulator, and a discriminator neural network. During reconstruction, the 3D structure is optimized so that its projections obtained through the simulator resemble real data (to the discriminator). Simultaneously, the discriminator is trained to distinguish real projections from simulated projections. CryoGAN takes as input only real projection images and the distribution of the cryo-EM imaging parameters. It involves neither prior training nor an initial estimation of the 3D structure. CryoGAN currently achieves a 10.8 Å resolution on a realistic synthetic dataset. Preliminary results on experimental β-galactosidase and 80S ribosome data demonstrate the ability of CryoGAN to exploit data statistics under standard experimental imaging conditions. We believe that this paradigm opens the door to a family of novel likelihood-free algorithms for cryo-EM reconstruction.
3 December 2019 in Foundations and Trends® in Signal Processing
This tutorial covers biomedical image reconstruction, from the foundational concepts of system modeling and direct reconstruction to modern sparsity and learning-based approaches. Imaging is a critical tool in biological research and medicine, and most imaging systems necessarily use an image reconstruction algorithm to create an image; the design of these algorithms has been a topic of research since at least the 1960’s. In the last few years, machine learning-based approaches have shown impressive performance on image reconstruction problems, triggering a wave of enthusiasm and creativity around the paradigm of learning. Our goal is to unify this body of research, identifying common principles and reusable building blocks across decades and among diverse imaging modalities. We first describe system modeling, emphasizing how a few building blocks can be used to describe a broad range of imaging modalities. We then discuss reconstruction algorithms, grouping them into three broad generations. The first are the classical direct methods, including Tikhonov regularization; the second are the variational methods based on sparsity and the theory of compressive sensing; and the third are the learning-based (also called data-driven) methods, especially those using deep convolutional neural networks. There are strong links between these generations: classical (first-generation) methods appear as modules inside the latter two, and the former two are used to inspire new designs for learning-based (third-generation) methods. As a result, a solid understanding of all three generations is necessary for the design of state-of-the-art algorithms.
28 October 2022 in Journal of Open Source Software