A support vector machine (SVM)-based classification nonlinear equalizer (NLE) is demonstrated, for the first time, in coherent optical orthogonal frequency-division multiplexing (CO-OFDM). For a 40-Gb/s 16 quadrature amplitude modulated (16QAM) CO-OFDM system at 400 km of transmission, SVM-NLE reduces the fiber-induced nonlinearity penalty by about 1 dB in comparison to the benchmark artificial-neural-network (ANN)-based and inverse-Volterra-series-transfer-function-based NLEs.

A novel versatile digital signal processing (DSP)-based equalizer using support vector machine regression (SVR) is proposed for 16-quadrature amplitude modulated (16-QAM) coherent optical orthogonal frequency-division multiplexing (CO-OFDM) and experimentally compared to traditional DSP-based deterministic fiber-induced nonlinearity equalizers (NLEs), namely the full-field digital back-propagation (DBP) and the inverse Volterra series transfer function-based NLE (V-NLE). For a 40 Gb/s 16-QAM CO-OFDM at 2000 km, SVR-NLE extends the optimum launched optical power (LOP) by 4 dB compared to V-NLE by means of reduction of fiber nonlinearity. In comparison to full-field DBP at a LOP of 6 dBm, SVR-NLE outperforms by $\sim 1\mathrm{dB}$ in $Q$-factor. In addition, SVR-NLE is the most computational efficient DSP-NLE.

In this paper, we experimentally demonstrate the combined benefit of artificial neural network-based nonlinearity compensation and probabilistic shaping for the first time. We demonstrate that the scheme not only compensates for transceiver's nonlinearity, enabling the full benefits of shaping to be achieved, but also the combined effects of transceiver and fiber propagation nonlinearities. The performance of the proposed artificial neural network is demonstrated at 28 Gbaud for both 64-QAM and 256-QAM probabilistically shaped systems and compared to that of uniformly distributed constellations. Our experimental results demonstrate: the expected performance gains for shaping alone; an additional SNR performance gain up to 1 dB in the linear region; an additional mutual information gain of 0.2 bits per channel use in the constellation-entropy limited region. In the presence of coupled transceiver and fiber-induced nonlinearities, an additional mutual information enhancement of ~0.13 bits/symbol is experimentally observed for a fiber link of up to 500 km with the aid of the proposed artificial neural network.

In this paper, we show the remarkable timing error (TE) and residual chromatic dispersion (CD) tolerance improvements of the filter bank multicarrier (FBMC) over orthogonal frequency division multiplexing (OFDM) for high-baudrate spectral slicing transmitter and single coherent receiver transmissions. For a 512 Gb/s 16 quadrature amplitude modulated (16QAM) spectrum slicing system at 1600 km of fiber transmission, the FBMC-based system reduces TE and residual CD penalties by more than 1.5 dB and 3 dB, in comparison to the OFDM-based system, respectively.

We propose a flexible simplified extended Kalman filter (S-EKF) scheme that can be applied in both pilot-aided and blind modes for phase noise compensation in 16-QAM CO-OFDM transmission systems employing a small-to-moderate number of subcarriers. The performance of the proposed algorithm is evaluated and compared with conventional pilot-aided (PA) and blind phase search (BPS) methods via extensive an Monte Carlo simulation in a back-to-back configuration and with a dual polarization fiber transmission. For 64 subcarrier 32 Gbaud 16-QAM CO-OFDM systems with 200 kHz combined laser linewidths, an optical signal-to-noise ratio penalty as low as 1 dB can be achieved with the proposed S-EKF scheme using only 2 pilots in the pilot-aided mode and just 4 inputs in the blind mode, resulting in a spectrally efficient enhancement by a factor of 3 and a computational effort reduction by a factor of more than 50 in comparison with the conventional PA and the BPS methods, respectively.

We propose a cost-efficient scheme called pseudo-pilot coding based phase noise estimation (PC-PNE) to compensate laser phase noises for filter-bank multicarrier (FBMC) transmission systems. The performance of the proposed PC-PNE with rectangular quadratic-amplitude modulation (QAM) is evaluated and compared with the state-of-the-art modified blind phase search (M-BPS) approach via extensive Monte Carlo simulations. Under 200 kHz combined laser linewidth, 32 Gbaud 256-subcarriers FBMC optical transmission equipped with PC-PNE provides a comparable performance with the complicated M-BPS method for QPSK, 16QAM and 64QAM. Moreover, a significant reduction in computational complexity (up to 50 times) is obtained with PC-PNE in comparison with M-BPS method at the cost of 3.1% spectral efficiency loss.