With the advances in scientific foundations and technological implementations, optical metrology has become versatile problem-solving backbones in manufacturing, fundamental research, and engineering applications, such as quality control, nondestructive testing, experimental mechanics, and biomedicine. In recent years, deep learning, a subfield of machine learning, is emerging as a powerful tool to address problems by learning from data, largely driven by the availability of massive datasets, enhanced computational power, fast data storage, and novel training algorithms for the deep neural network. It is currently promoting increased interests and gaining extensive attention for its utilization in the field of optical metrology. Unlike the traditional “physics-based” approach, deep-learning-enabled optical metrology is a kind of “data-driven” approach, which has already provided numerous alternative solutions to many challenging problems in this field with better performances. In this review, we present an overview of the current status and the latest progress of deep-learning technologies in the field of optical metrology. We first briefly introduce both traditional image-processing algorithms in optical metrology and the basic concepts of deep learning, followed by a comprehensive review of its applications in various optical metrology tasks, such as fringe denoising, phase retrieval, phase unwrapping, subset correlation, and error compensation. The open challenges faced by the current deep-learning approach in optical metrology are then discussed. Finally, the directions for future research are outlined.

In many optical metrology techniques, fringe pattern analysis is the central algorithm for recovering the underlying phase distribution from the recorded fringe patterns. Despite extensive research efforts for decades, how to extract the desired phase information, with the highest possible accuracy, from the minimum number of fringe patterns remains one of the most challenging open problems. Inspired by recent successes of deep learning techniques for computer vision and other applications, we demonstrate for the first time, to our knowledge, that the deep neural networks can be trained to perform fringe analysis, which substantially enhances the accuracy of phase demodulation from a single fringe pattern. The effectiveness of the proposed method is experimentally verified using carrier fringe patterns under the scenario of fringe projection profilometry. Experimental results demonstrate its superior performance, in terms of high accuracy and edge-preserving, over two representative single-frame techniques: Fourier transform profilometry and windowed Fourier transform profilometry. Optical measurement techniques such as holographic interferometry,¹ electronic speckle pattern interferometry,² and fringe projection profilometry³ are quite popular for noncontact measurements in many areas of science and engineering, and have been extensively applied for measuring various physical quantities, such as displacement, strain, surface profile, and refractive index. In all these techniques, the information about the measured physical quantity is stored in the phase of a two-dimensional fringe pattern. The accuracy of measurements carried out by these optical techniques is thus fundamentally dependent on the accuracy with which the underlying phase distribution of the recorded fringe patterns is demodulated. Over the past few decades, tremendous efforts have been devoted to developing various techniques for fringe analysis. The techniques can be broadly classified into two categories: (1) phase-shifting (PS) methods that require multiple fringe patterns to extract phase information,⁴ and (2) spatial phase-demodulation methods that allow phase retrieval from a single fringe pattern, such as the Fourier transform (FT),⁵ windowed Fourier transform (WFT),⁶ and wavelet transform (WT) methods.⁷ Compared with spatial phase demodulation methods, multiple-shot PS techniques are generally more robust and can achieve pixel-wise phase measurement with higher resolution and accuracy. Furthermore, the PS measurements are quite insensitive to nonuniform background intensity and fringe modulation. Nevertheless, due to their multishot nature, these methods are difficult to apply to dynamic measurements and are more susceptible to external disturbance and vibration. Thus, for many applications, phase extraction from a single fringe pattern is desired, which falls under the purview of spatial fringe analysis. In contrast to PS techniques where the phase map is demodulated on a pixel-by-pixel basis, phase estimation at a pixel according to spatial methods is influenced by the pixel’s neighborhood, or even all pixels in the fringe pattern, which provides better tolerance to noise, yet at the expense of poor performance around discontinuities and isolated regions in the phase map.^8,9 Deep learning is a powerful machine learning technique that employs artificial neural networks with multiple layers of increasingly richer functionality and has shown great success in numerous applications for which data are abundant.^10,11 In this letter, we demonstrate experimentally for the first time, to our knowledge, that the use of a deep neural network can substantially enhance the accuracy of phase demodulation from a single fringe pattern. To be concrete, the networks are trained to predict several intermediate results that are useful for the calculation of the phase of an input fringe pattern. During the training of the networks, we capture PS fringe images of various scenes to generate the training data. The training label (ground truth) of each training datum is a pair of intermediate results calculated from the PS algorithm. After appropriate training, the neural network can blindly take only one input fringe pattern and output the corresponding estimates of these intermediate results with high fidelity. Finally, a high-accuracy phase map can be retrieved through the arctangent function with the intermediate results estimated through deep learning. Experimental results on fringe projection profilometry confirm that this deep-learning-based method is able to substantially improve the quality of the retrieved phase from only a single fringe pattern, compared to state-of-the-art methods. Here, the network configuration is inspired by the basic process of most phase demodulation techniques, which is briefly recalled as follows. The mathematical form of a typical fringe pattern can be represented as <!-- named anchor --> (1)$$I(x,y)=A(x,y)+B(x,y)\cos \varphi (x,y),$$where $I(x,y)$ is the intensity of the fringe pattern, $A(x,y)$ is the background intensity, $B(x,y)$ is the fringe amplitude, and $\varphi (x,y)$ is the desired phase distribution. Here, $x$ and $y$ refer to the pixel coordinates. In most phase demodulation techniques, the background intensity $A(x,y)$ is regarded as a disturbance term and should be removed from the total intensity. Then a wrapped phase map is recovered from an inverse trigonometric function whose argument is a ratio for which the numerator characterizes the phase sine [$\sin \varphi (x,y)$] and the denominator characterizes the phase cosine [$\cos \varphi (x,y)$]: <!-- named anchor --> (2)$$\varphi (x,y)=\arctan \frac{M(x,y)}{D(x,y)}=\arctan \frac{cB(x,y)\sin \varphi (x,y)}{cB(x,y)\cos \varphi (x,y)},$$where $c$ is a constant dependent on the phase demodulation algorithm (e.g., in FT $c=0.5$, in $N$-step PS $c=N/2$), and $M(x,y)$ and $D(x,y)$ represent the shorthand for the numerator and denominator terms, respectively. Note that the signs of $M(x,y)$ and $D(x,y)$ can be further used to uniquely define a quadrant for each calculation of $\varphi (x,y)$. With the four-quadrant phasor space, the phase values at each point can be determined modulo $2\pi$.In order to emulate the process above, two different convolutional neural networks (CNN) are constructed, which are connected cascadedly according to Fig. 1. The first convolutional neural network (CNN1) uses the raw fringe pattern $I(x,y)$ as input and estimates the background intensity $A(x,y)$ of the fringe pattern. With the estimated background image $A(x,y)$ and the original fringe image $I(x,y)$, the second convolutional neural network (CNN2) is trained to predict the numerator $M(x,y)$ and the denominator $D(x,y)$ of the arctangent function, which are fed into the subsequent arctangent function [Eq. (2)] to obtain the final phase distribution $\varphi (x,y)$. <!-- named anchor --> Fig. 1 Flowchart of the proposed method where two convolutional networks (CNN1 and CNN2) and the arctangent function are used together to determine the phase distribution. For CNN1 (in red), the input is the fringe image $I(x,y)$, and the output is the estimated background image $A(x,y)$. For CNN2 (in green), the inputs are the fringe image $I(x,y)$ and the background image $A(x,y)$ predicted by CNN1, and the outputs are the numerator $M(x,y)$ and the denominator $D(x,y)$. The numerator and denominator are then fed into the arctangent function to calculate the phase $\varphi (x,y)$. To generate the ground truth data used as the label to train the two convolutional neural networks, the phase retrieval is achieved by using the $N$-step PS method. The corresponding $N$ PS fringe patterns acquired can be represented as <!-- named anchor --> (3)$$I_n(x,y)=A(x,y)+B(x,y)\cos [\varphi (x,y)-{\delta }_n],$$where the index $n=0,1,\dots ,N-1$, and ${\delta }_n$ is the phase shift that equals $\frac{2\pi n}{N}$. With the orthogonality of trigonometric functions, the background intensity can be obtained as <!-- named anchor --> (4)$$A(x,y)=\frac{1}{N}\sum _{n=0}^{N-1}{I}_n(x,y).$$ With the least square method, the phase can be calculated as <!-- named anchor --> (5)$$\varphi (x,y)=\arctan \frac{\sum _{n=0}^{N-1}{I}_n(x,y)\sin {\delta }_n}{\sum _{n=0}^{N-1}{I}_n(x,y)\cos {\delta }_n}.$$ Thus, the numerator and the denominator of the arctangent function in Eq. (2) can be expressed as <!-- named anchor --> (6)$$M(x,y)=\sum _{n=1}^{N-1}{I}_n(x,y)\sin {\delta }_n=\frac{N}{2}B(x,y)\sin \varphi (x,y),$$ <!-- named anchor --> (7)$$D(x,y)=\sum _{n=0}^{N-1}{I}_n(x,y)\cos {\delta }_n=\frac{N}{2}B(x,y)\cos \varphi (x,y).$$ The expressions above show that the numerator $M(x,y)$ and the denominator $D(x,y)$ are closely related to the original fringe pattern in Eq. (1) through a quasilinear relationship with the background image