This paper reviews digital signal processing techniques that compensate, mitigate, and exploit fiber nonlinearities in coherent optical fiber transmission systems.

The nonlinear Fourier transform is a new approach of addressing the capacity limiting Kerr nonlinearities in optical communication systems. It exploits the property of integrability of the lossless nonlinear Schrödinger equation and thus incorporates nonlinearities as an element of the transmission. However, practical links employing erbium-doped fiber amplifiers include losses/gains and introduce noise which breaks the integrability of the nonlinear Schrödinger equation. Although the lossless path average approximation proposes an integrable model, its imprecision still leads to unintended distortions and thus performance degradation. We propose an alternative receiver for nonlinear frequency division multiplexing optical communication systems using techniques from machine learning. It is highly adaptive as it learns from previously transmitted pulses and thus holds no assumptions on the system and noise distribution. The detection method presented is fully applied in time-domain and omits the nonlinear Fourier transform. The numerical results provide a benchmark for nonlinear Fourier transform based detection of high order solitons for fiber links with losses and noise present.

Training of autoencoders using the back-propagation algorithm is challenging for non-differential channel models or in an experimental environment where gradients cannot be computed. In this paper, we study a gradient–free training method based on the cubature Kalman filter. To numerically validate the method, the autoencoder is employed to perform geometric constellation shaping on differentiable communication channels, showing the same performance as the back-propagation algorithm. Further investigation is done on a non–differentiable communication channel that includes: laser phase noise, additive white Gaussian noise and blind phase search-based phase noise compensation. Our results indicate that the autoencoder can be successfully optimized using the proposed training method to achieve better robustness to residual phase noise with respect to standard constellation schemes such as Quadrature Amplitude Modulation and Iterative Polar Modulation for the considered conditions.

End-to-end learning has become a popular method to optimize a constellation shape of a communication system. When the channel model is differentiable, end-to-end learning can be applied with conventional backpropagation algorithm for optimization of the shape. A variety of optimization algorithms have also been developed for end-to-end learning over a non-differentiable channel model. In this paper, we compare a gradient-free optimization method based on the cubature Kalman filter, model-free optimization and backpropagation for end-to-end learning on a fiber-optic channel modeled by the split-step Fourier method. The results indicate that the gradient-free optimization algorithms provide a decent replacement to backpropagation in terms of performance at the expense of computational complexity. Furthermore, the quantization problem of finite bit resolution of the digital-to-analog and analog-to-digital converters is addressed and its impact on geometrically shaped constellations is analysed. Here, the results show that when optimizing a constellation with respect to mutual information, a minimum number of quantization levels is required to achieve shaping gain. For generalized mutual information, the gain is maintained throughout all of the considered quantization levels. Also, the results imply that the autoencoder can adapt the constellation size to the given channel conditions.

Vendor interoperability is one of the desired future characteristics of optical networks. This means that the transmission system needs to support a variety of hardware with different components, leading to system uncertainties throughout the network. For example, uncertainties in signal-to-noise ratio and laser linewidth can negatively affect the quality of transmission within an optical network due to e.g. mis-parametrization of the transceiver signal processing algorithms. In this paper, we propose to geometrically optimize a constellation shape that is robust to uncertainties in the channel conditions by utilizing end-to-end learning. In the optimization step, the channel model includes additive noise and residual phase noise. In the testing step, the channel model consists of laser phase noise, additive noise and blind phase search as the carrier phase recovery algorithm. Two noise models are considered for the additive noise: white Gaussian noise and nonlinear interference noise model for fiber nonlinearities. The latter models the behavior of an optical fiber channel more accurately because it considers the nonlinear effects of the optical fiber. For this model, the uncertainty in the signal-to-noise ratio can be divided between amplifier noise figures and launch power variations. For both noise models, our results indicate that the learned constellations are more robust to the uncertainties in channel conditions compared to a standard constellation scheme such as quadrature amplitude modulation and standard geometric constellation shaping techniques.