In this paper, we provide an overview of recent progress on advanced digital signal processing (DSP) techniques for high-capacity long-haul coherent optical fiber transmission systems. Not only the linear impairments existing in optical transmission links need to be compensated, but also, the nonlinear impairments require proper algorithms for mitigation because they become major limiting factors for long-haul large-capacity optical transmission systems. Besides the time domain equalization (TDE), the frequency domain equalization (FDE) DSP also provides a similar performance, with a much-reduced computational complexity. Advanced DSP also plays an important role for the realization of space division multiplexing (SDM). SDM techniques have been developed recently to enhance the system capacity by at least one order of magnitude. Some impressive results have been reported and have outperformed the nonlinear Shannon limit of the single-mode fiber (SMF). SDM introduces the space dimension to the optical fiber communication. The few-mode fiber (FMF) and multi-core fiber (MCF) have been manufactured for novel multiplexing techniques such as mode-division multiplexing (MDM) and multi-core multiplexing (MCM). Each mode or core can be considered as an independent degree of freedom, but unfortunately, signals will suffer serious coupling during the propagation. Multi-input–multi-output (MIMO) DSP can equalize the signal coupling and makes SDM transmission feasible. The machine learning (ML) technique has attracted worldwide attention and has been explored for advanced DSP. In this paper, we firstly introduce the principle and scheme of coherent detection to explain why the DSP techniques can compensate for transmission impairments. Then corresponding technologies related to the DSP, such as nonlinearity compensation, FDE, SDM and ML will be discussed. Relevant techniques will be analyzed, and representational results and experimental verifications will be demonstrated. In the end, a brief conclusion and perspective will be provided.

In this work, a perturbation-based neural network (P-NN) scheme with an embedded bidirectional long short-term memory (biLSTM) layer is investigated to compensate for the Kerr fiber nonlinearity in optical fiber communication systems. Numerical simulations have been carried out in a 32-Gbaud dual-polarization 16-ary quadrature amplitude modulation (DP-16QAM) transmission system. It is shown that this P-NN equalizer can achieve signal-to-noise ratio improvements of ∼1.37 dB and ∼0.80 dB, compared to the use of a linear equalizer and a single step per span (StPS) digital back propagation (DBP) scheme, respectively. The P-NN equalizer requires lower computational complexity and can effectively compensate for intra-channel nonlinearity. Meanwhile, the performance of P-NN is more robust to the distortion caused by equalization enhanced phase noise (EEPN). Furthermore, it is also found that there exists a tradeoff between the choice of modulation format and the nonlinear equalization schemes for a given transmission distance.

6 × 10 Gb/s polarization- and mode-multiplexed transmission, without MIMO equalization, is experimentally demonstrated over a 200-m elliptical core few-mode fiber with two PANDA stress rods. This fiber simultaneously breaks the degeneracies in space and polarization in a mode group. A polarization extinction ratio of <;-16 dB and loss <;0.63 dB/km were achieved.

Polarization-maintaining few-mode fibers (PM-FMFs) have found applications in sensors with attractive features that traditional single-mode fibers do not possess. We propose a measurement method for multi-parameter sensing based on PM-FMFs. This method is mainly based on the polarized interference effect of PM-FMF. We experimentally demonstrate the principle of the simultaneous sensing of temperature, strain and transverse force. The sensor has been verified by three groups of multi-parameter measurement experiments showing that the changes of multi-parameter derived from the wavelength shift of the polarized interference spectrum are consistent with the multi-parameter changes set in the experiments. The proposed PM-FMF based sensor has a temperature sensitivity of 0.3 nm/°C, a strain sensitivity of 0.01 nm/µε and a transverse force sensitivity of 0.0065 nm/(N/m).

Simultaneous measurement of temperature and strain was demonstrated using a polarization-maintaining few-mode Bragg grating (PM-FMF-FBG) based on the wavelength and phase modulation of the even $L{P}_{11}$ mode. The wavelength shift sensitivity and the interrogated phase sensitivity of the temperature and strain were measured to be 10 pm·°C⁻¹ and 0.73 pm·με⁻¹ and −3.2 × 10⁻² rad·°C⁻¹ and 4 × 10⁻⁴ rad·με⁻¹, respectively, with a discrimination efficiency of 98%. The polarization interference led to selective polarization excitation of the reflection spectra, and the calculated phase sensitivity agreed with the experimental results.

We report the design of 12-LP-mode (21-spatial-mode) fiber with low differential mode group delay (DMGD), large effective area and low bending loss that is adapted to mode-division-multiplexing (MDM) system over the C+L band. Based on the designed fiber parameters, we characterize the few-mode fiber (FMF) with the DMGD, effective area and bending loss. Over the C+L band, the maximum DMGD is 0.106 ps/m, and the effective area is in the range of 150~485 μm2. The bending loss of the designed FMF reduces to zero when the bending radius is greater than 9 mm, and the DMGD is below 0.0413 ps/m at the wavelength of 1550 nm.