Nonlinear impairments caused by devices and fiber transmission links in a coherent optical communication system can severely limit its transmission distance and achievable capacity. In this paper, we propose a low-complexity pruned-convolutional-neural-network-(CNN)-based nonlinear equalizer, to compensate nonlinear signal impairments for coherent optical communication systems. By increasing the size of the effective receptive field with an 11 × 11 large convolutional kernel, the performance of feature extraction for CNNs is enhanced and the structure of the CNN is simplified. And by performing the channel-level pruning algorithm, to prune the insignificant channels, the complexity of the CNN model is dramatically reduced. These operations could save the important component of the CNN model and reduce the model width and computation amount. The performance of the proposed CNN-based nonlinear equalizer was experimentally evaluated in a 120 Gbit/s 64-quadrature-amplitude-modulation (64-QAM) coherent optical communication system over 375 km of standard single-mode fiber (SSMF). The experimental results showed that, compared to a CNN-based nonlinear equalizer with a 6 × 6 normal convolutional kernel, the proposed CNN-based nonlinear equalizer with an 11 × 11 large convolutional kernel, after channel-level pruning, saved approximately 15.5% space complexity and 43.1% time complexity, without degrading the equalization performance. The proposed low-complexity pruned-CNN-based nonlinear equalizer has great potential for application in realistic devices and holds promising prospects for coherent optical communication systems.

In this study, we developed a wide and deep network-based nonlinear equalizer to compensate for nonlinear impairment in coherent optical communication systems. In our proposed equalizer, the power feature factor and inter-symbol feature sequence in the received signal are analyzed by two combined networks, wide and deep, respectively, so that the information contained in the signal can be fully utilized. We designed an experiment using a 120 Gbit/s 64-quadrature amplitude modulation (64-QAM) coherent optical communication system over a 375 km standard single-mode fiber (SSMF) to verify the performance of the proposed wide and deep network-based nonlinear equalizer. The experimental results showed that the proposed wide and deep network-based nonlinear equalizer achieved better performance at lower complexity compared with the traditional neural network-based nonlinear equalizer. The proposed equalizer significantly improved the equalization effect at a cost of a 0.3% increase in parameters, which indicates the potential of the proposed method for application in coherent optical communication systems.