Practical implementation of digital signal processing for mitigation of transmission impairments in optical communication systems requires reduction of the complexity of the underlying algorithms. Here, we investigate the application of convolutional neural networks for compensating nonlinear signal distortions in a 3200 km fiber-optic 11x400-Gb/s WDM PDM-16QAM transmission link with a focus on the optimization of the corresponding algorithmic complexity. We propose a design that includes original initialisation of the weights of the layers by a filter predefined through the training a single-layer convolutional neural network. Furthermore, we use an enhanced activation function that takes into account nonlinear interactions between neighbouring symbols. To increase learning efficiency, we apply a layer-wise training scheme followed by joint optimization of all weights applying additional training to all of them together in the large multi-layer network. We examine application of the proposed convolutional neural network for the nonlinearity compensation using only one sample per symbol and evaluate complexity and performance of the proposed technique.

We propose a modification of the conventional perturbation-based approach of fiber nonlinearity compensation that enables straight-forward implementation at the receiver and meets feasible complexity requirements. We have developed a model based on perturbation analysis of an inverse Manakov problem, where we use the received signal as the initial condition and solve Manakov equations in the reversed direction, effectively implementing a perturbative digital backward propagation enhanced by machine learning techniques. To determine model coefficients we employ machine learning methods using a training set of transmitted symbols. The proposed approach allowed us to achieve 0.5 dB and 0.2 dB $Q^2$ -factor improvement for 2000 km transmission of 11x256 Gbit/s DP-16QAM signal compared to chromatic dispersion equalization and one step per span two samples per symbol digital back-propagation technique, respectively. We quantify the trade-off between performance and complexity.

The data transmission in coherent fiber-optical communication lines using solitons with a variable phase is studied. It is shown that nonlinear coherent structures (solitons) can be applied for...

We have revisited soliton transmission in the new context of coherent optical detection optimizing and comparing digital backward propagation and in-line optical filtering as a means to suppress soliton timing and phase jitter. We find that in-line optical filtering allows one to improve the reach of the soliton system by up to the factor of 2. Our results show that nonlinear propagation can lead to performance beyond the nonlinear Shannon limit.

We perform an extensive numerical analysis of Raman-Assisted Fibre Optical Parametric Amplifiers (RA-FOPA) in the context of WDM QPSK signal amplification. A detailed comparison of the conventional FOPA and RA-FOPA is reported and the important advantages offered by the Raman pumping are clarified. We assess the impact of pump power ratios, channel count, and highly nonlinear fibre (HNLF) length on crosstalk levels at different amplifier gains. We show that for a fixed 200 m HNLF length, maximum crosstalk can be reduced by up to 7 dB when amplifying 10x58Gb/s QPSK signals at 20 dB net-gain using a Raman pump of 37 dBm and parametric pump of 28.5 dBm in comparison to a standard single-pump FOPA using 33.4 dBm pump power. It is shown that a significant reduction in four-wave mixing crosstalk is also obtained by reducing the highly nonlinear fibre interaction length. The trend is shown to be generally valid for different net-gain conditions and channel grid size. Crosstalk levels are additionally shown to strongly depend on the Raman/parametric pump power ratio, with a reduction in crosstalk seen for increased Raman pump power contribution.