Channel modeling plays a pivotal role in the field of communications, particularly in the optical communication networks of backbone communication systems. Recent studies on optical channel modeling have utilized real-valued neural network (RVNN) to extract channel characteristics, an approach that does not fully account for the properties of complex-valued signals. To address this limitation, we propose a complex-valued conditional generative adversarial network (C-CGAN) in this paper to comprehensively learn channel features. We describe the architecture and parameters of the C-CGAN and employ complex-valued windowed construction for input data. Subsequently, we evaluate the model’s accuracy and generalization capabilities using the normalized mean square error (NMSE) and benchmark it against the real-valued conditional generative adversarial network (R-CGAN). The results indicate that the C-CGAN achieves better generalization across various scenarios, including different dataset sizes, noise levels, and input feature complexities, while also exhibiting a more stable training process. The NMSE achieved by the C-CGAN remains below \(2\times 10^{-2}\) and outperforms the R-CGAN. Additionally, analysis from the perspective of floating-point operations (FLOPs) reveals that the computational complexity of the C-CGAN is relatively low. To further validate scalability, we introduce a self-loop cascading mechanism that, under constrained training datasets, improves NMSE performance by 22.48% compared to the R-CGAN.

In this paper, an improved complex-valued convolutional neural network (CvCNN) structure to be placed at the received side is proposed for nonlinearity compensation in a coherent optical system. This complex-valued global convolutional kernel-assisted convolutional neural network equalizer (CvGNN) has been verified in terms of Q-factor performance and complexity compared to seven other related nonlinear equalizers based on both the 64 QAM experimental platform and the QPSK numerical platform. The global convolution operation of the proposed CvGNN is more suitable for the calculation process of perturbation coefficients, and the global receptive field can also be more effective at extracting effective information from perturbation feature maps. The introduction of CvCNN can directly focus on the complex-valued perturbation feature maps themselves without separately processing the real and imaginary parts, which is more in line with the waveform-dependent physical characteristics of optical signals. Based on the experimental platform, compared with the real-valued neural network with small convolutional kernel (RvCNNC), the proposed CvGNNC improves the Q-factor by ∼2.95 dB at the optimal transmission power, while reducing the time complexity by ∼44.7%.