We present the results of the comparative performance-versus-complexity analysis for the several types of artificial neural networks (NNs) used for nonlinear channel equalization in coherent optical communication systems. The comparison is carried out using an experimental set-up with the transmission dominated by the Kerr nonlinearity and component imperfections. For the first time, we investigate the application to the channel equalization of the convolution layer (CNN) in combination with a bidirectional long short-term memory (biLSTM) layer and the design combining CNN with a multi-layer perceptron. Their performance is compared with the one delivered by the previously proposed NN-based equalizers: one biLSTM layer, three-dense-layer perceptron, and the echo state network. Importantly, all architectures have been initially optimized by a Bayesian optimizer. First, we present the general expressions for the computational complexity associated with each NN type; these are given in terms of real multiplications per symbol. We demonstrate that in the experimental system considered, the convolutional layer coupled with the biLSTM (CNN+biLSTM) provides the largest Q-factor improvement compared to the reference linear chromatic dispersion compensation (2.9 dB improvement). Then, we examine the trade-off between the computational complexity and performance of all equalizers and demonstrate that the CNN+biLSTM is the best option when the computational complexity is not constrained, while when we restrict the complexity to some lower levels, the three-layer perceptron provides the best performance.

This paper performs a detailed, multi-faceted analysis of key challenges and common design caveats related to the development of efficient neural networks (NN) based nonlinear channel equalizers in coherent optical communication systems. The goal of this study is to guide researchers and engineers working in this field. We start by clarifying the metrics used to evaluate the equalizers’ performance, relating them to the loss functions employed in the training of the NN equalizers. The relationships between the channel propagation model’s accuracy and the performance of the equalizers are addressed and quantified. Next, we assess the impact of the order of the pseudo-random bit sequence used to generate the – numerical and experimental – data as well as of the DAC memory limitations on the operation of the NN equalizers both during the training and validation phases. Finally, we examine the critical issues of overfitting limitations, the difference between using classification instead of regression, and batch-size-related peculiarities. We conclude by providing analytical expressions for the equalizers’ complexity evaluation in the digital signal processing (DSP) terms and relate the metrics to the processing latency.

Experimental results based on offline processing reported at optical conferences increasingly rely on neural network-based equalizers for accurate data recovery. However, achieving low-complexity implementations that are efficient for real-time digital signal processing remains a challenge. This paper addresses this critical need by proposing a systematic approach to designing and evaluating low-complexity neural network equalizers. Our approach focuses on three key phases: training, inference, and hardware synthesis. We provide a comprehensive review of existing methods for reducing complexity in each phase, enabling informed choices during design. For the training and inference phases, we introduce a novel methodology for quantifying complexity. This includes new metrics that bridge software-to-hardware considerations, revealing the relationship between complexity and specific neural network architectures and hyperparameters. We guide the calculation of these metrics for both feed-forward and recurrent layers, highlighting the appropriate choice depending on the application's focus (software or hardware). Finally, to demonstrate the practical benefits of our approach, we showcase how the computational complexity of neural network equalizers can be significantly reduced and measured for both teacher (biLSTM+CNN) and student (1D-CNN) architectures in different scenarios. This work aims to standardize the estimation and optimization of computational complexity for neural networks applied to real-time digital signal processing, paving the way for more efficient and deployable optical communication systems.

Nonlinearity compensation is considered as a key enabler to increase channel transmission rates in the installed optical communication systems. Recently, data-driven approaches — motivated by modern machine learning techniques — have been proposed for optical communications in place of traditional model-based counterparts. In particular, the application of neural networks (NN) allows improving the performance of complex modern fiber-optic systems without relying on any a priori knowledge of their specific parameters. In this work, we introduce a novel design of complex-valued NN for optical systems and examine its performance in standard single mode fiber (SSMF) and large effective-area fiber (LEAF) links operating in relatively high nonlinear regime. First, we present a methodology to design a new type of NN based on the assumption that the channel model is more accurate in the nonlinear regime. Second, we implement a Bayesian optimizer to jointly adapt the size of the NN and its number of input taps depending on the different fiber properties and total length. Finally, the proposed NN is numerically and experimentally validated showing an improvement of 1.7 dB in the linear regime, 2.04 dB at the optimal optical power and 2.61 at the max available power on Q-factor when transmitting a WDM 30×200G DP-16QAM signal over a 620 km SSMF legacy link. The results highlight that the NN is able to mitigate not only part of the nonlinear impairments caused by optical fiber propagation but also imperfections resulting from using low-cost legacy transceiver components, such as digital-to-analog converter (DAC) and Mach-Zehnder modulator.

In this work, we address the question of the adaptability of artificial neural networks (NNs) used for impairments mitigation in optical transmission systems. We demonstrate that by using well-developed techniques based on the concept of transfer learning, we can efficaciously retrain NN-based equalizers to adapt to the changes in the transmission system, using just a fraction (down to 1%) of the initial training data or epochs. We evaluate the capability of transfer learning to adapt the NN to changes in the launch power, modulation format, symbol rate, or even fiber plants (different fiber types and lengths). The numerical examples utilize the recently introduced NN equalizer consisting of a convolutional layer coupled with bi-directional long-short term memory (biLSTM) recurrent NN element. Our analysis focuses on long-haul coherent optical transmission systems for two types of fibers: the standard single-mode fiber (SSMF) and the TrueWave Classic (TWC) fiber. We underline the specific peculiarities that occur when transferring the learning in coherent optical communication systems and draw the limits for the transfer learning efficiency. Our results demonstrate the effectiveness of transfer learning for the fast adaptation of NN architectures to different transmission regimes and scenarios, paving the way for engineering flexible and universal solutions for nonlinearity mitigation.

In this paper, a new methodology is proposed that allows for the low-complexity development of neural network (NN) based equalizers for the mitigation of impairments in high-speed coherent optical transmission systems. In this work, we provide a comprehensive description and comparison of various deep model compression approaches that have been applied to feed-forward and recurrent NN designs. Additionally, we evaluate the influence these strategies have on the performance of each NN equalizer. Quantization, weight clustering, pruning, and other cutting-edge strategies for model compression are taken into consideration. In this work, we propose and evaluate a Bayesian optimization-assisted compression, in which the hyperparameters of the compression are chosen to simultaneously reduce complexity and improve performance. Next, this paper presents four distinct metrics (RMpS, BoP, NABS, and NLGs) that are discussed here that can be used to evaluate the amount of computing complexity required by various compression algorithms. These measurements can serve as a benchmark for evaluating the relative effectiveness of various NN equalizers when compression approaches are used. In conclusion, the trade-off between the complexity of each compression approach and its performance is evaluated by utilizing both simulated and experimental data in order to complete the analysis. By utilizing optimal compression approaches, we show that it is possible to design an NN-based equalizer that is simpler to implement and has better performance than the conventional digital back-propagation (DBP) equalizer with only one step per span. This is accomplished by reducing the number of multipliers used in the NN equalizer after applying the weighted clustering and pruning algorithms. Furthermore, we demonstrate that an equalizer based on NN can also achieve superior performance while still maintaining the same degree of complexity as the full electronic chromatic dispersion compensation block. We conclude our analysis by highlighting open questions and existing challenges, as well as possible future research directions.