Atmospheric turbulence causes signal beam wavefront distortion at the receiving end of a coherent detection system, which decreases the system mixing efficiency. Based on the coherent detection theory, this study establishes a mathematical model of wavefront distortion with mixing efficiency and mixing gain. It also analyzes the improvement limits of wavefront correction on mixing efficiency and mixing gain under different atmospheric turbulence intensities and experimentally measures them. Simulation results show that the mixing efficiency can be improved to 51%, 55%, and 60% after correcting for tilt, defocus, and astigmatism terms, respectively, when turbulence intensity D/r₀ is 2. The mixing gain with homodyne detection is 3 dB higher than heterodyne detection. Meanwhile, the wavefront correction orders required for optimal mixing efficiency are higher than the heterodyne correction order. In the experiment, Haso4 NIR + DM 40 was used, and the turbulence intensity D/r₀ was 2. After the closed-loop control algorithm corrects the tilt, defocus, and astigmatism terms, the indoor experimental results showed that the mixing efficiency is improved to 36%, 47%, and 62%, respectively. The outdoor experimental results showed that the mixing efficiency improved to 36%, 51%, and 68%, respectively.

This study aims to address the multi-beam transmission problem of optical wireless coherent communication systems under the influence of non-Kolmogorov turbulence. This paper establishes the mathematical model for mixing efficiency and BER of non-Kolmogorov turbulence wavefront distortion and multi-beam transmission coherent detection systems. The influence of factors such as spectral power-law index, zenith angle, and transmission distance on the communication performance of the system is analyzed, and an experimental system is built using adaptive optics equipment to test the proposed theory. Numerical analyses show that in a non-Kolmogorov turbulence environment, as the spectral power-law index, zenith angle, transmission distance, Fresnel zone, and pointing error attenuation increase, the mixing efficiency of the coherent detection system decreases and the BER increases; under the same conditions, the mixing efficiency and BER of the system are improved after increasing the number of transmission beams at the transmitting end. Experimental research shows that, for the coherent detection system after wavefront correction using adaptive optics, the wavefront PV and RMS variances are 1.68 μm² and 0.05 μm² when a single beam of light is transmitted at the transmitting end, and the wavefront PV and RMS variances are 0.23 μm² and 0.01 μm² when two beams are transmitted. It is verified that using multi-beam transmission wavefront superposition to suppress wavefront distortion and wavefront jitter can effectively improve the mixing efficiency and BER of the coherent detection system at the receiving end.