Even after the Heydemann correction, residual nonlinear errors, ranging from hundreds of picometers to several nanometers, are still found in heterodyne laser interferometers. This is a crucial factor impeding the realization of picometer level metrology, but its source and mechanism have barely been investigated. To study this problem, a novel nonlinear model based on optical mixing and coupling with ghost reflection is proposed and then verified by experiments. After intense investigation of this new model’s influence, results indicate that new additional high-order and negative-order nonlinear harmonics, arising from ghost reflection and its coupling with optical mixing, have only a negligible contribution to the overall nonlinear error. In real applications, any effect on the Lissajous trajectory might be invisible due to the small ghost reflectance. However, even a tiny ghost reflection can significantly worsen the effectiveness of the Heydemann correction, or even make this correction completely ineffective, i.e., compensation makes the error larger rather than smaller. Moreover, the residual nonlinear error after correction is dominated only by ghost reflectance.

The phase quadrature measurement method is capable of measuring nonlinearity in heterodyne laser interferometers with picometer accuracy whereas it cannot be applied in the new kind of heterodyne interferometers with bidirectional Doppler frequency shift especially in the condition of non-uniform motion of the target. To solve this problem, a novel measurement method of nonlinearity is proposed in this paper. By employing double-channel quadrature demodulation and substituting the external reference signal with internal ones, this method is free from the type of heterodyne laser interferometer and the motion state of the target. For phase demodulation, the phase differential algorithm is utilized to improve the computing efficiency. Experimental verification is carried out and the results indicate that the proposed measurement method achieves accuracy better than 2 pm.