In-band crosstalk, due to multiple interferers, has been identified as one of the most severe impairments in optical transparent networks, especially in the ones with a large number of nodes and a high wavelength density. Due to its robustness to in-band crosstalk differential phase-shift keying (DPSK) emerges as an attractive modulation scheme to be used in such environments. This paper proposes a rigorous formulation to estimate the performance of direct-detection DPSK receivers using an eigenfunction expansion technique in the time domain. The method takes into account both the in-band crosstalk, due to an arbitrary number of interfering terms, and the amplified spontaneous emission noise and is able to deal with any combination of optical and electrical filter shapes. Using this method the accuracy of an approximation based on the wideband optical filtering assumption was evaluated and shown that the approximation, although not providing reliable results for the error probabilities, can be used with confidence to compute power penalties due to in-band crosstalk. Furthermore, the crosstalk tolerance of DPSK over on-off keying was quantified and shown that this tolerance is reduced when the number of interferers increases.

Optical coherent technology leads to the coexistence of signals with different modulation formats and different bit rates in optical networks. Due to this coexistence, several in-band crosstalk scenarios are possible. In this work, the impact of in-band crosstalk due to M-QAM interferers on the performance of PDM-QPSK and PDM-16-QAM coherent receivers at a bit rate of 100 Gbps is studied. The influence of the duty-cycle, time misalignment and phase difference between interferers and selected signal on the receiver performance is also assessed. We show that the QPSK signal with a duty-cycle of 33% is, generally, the most tolerant selected signal to in-band crosstalk and the less detrimental interferer to the coherent receiver performance. Additionally, we show that the time misalignment has a strong influence on the in-band crosstalk impact, when the duty-cycles of the interferers or selected signal are low.

This paper provides new theoretical insights into the properties of direct detection differential phase-shift keying (DPSK) signals impaired by coherent and incoherent crosstalk. Coherent crosstalk is due to multiple replicas originated while a data signal is routed through an optical network, whereas the source of incoherent crosstalk resides on the interference from other DPSK signals. A special emphasize has been put on modeling the multipath coherent crosstalk, with analytical expressions being derived and presented for both the moment generating function of the decision variable and the average error probability. A rigorous analysis, capable of dealing with arbitrary filtering, is also presented, which is used afterwards to assess the accuracy of the analytical formulas. A detailed comparison with incoherent crosstalk is also performed. Using also an exact treatment for this type of crosstalk it is shown that for low OSNR penalties the coherent crosstalk leads, in some circumstances, to slightly worse results than the incoherent one, but this situation is reversed when the total crosstalk level and the number of interferers increase.

An analytical formalism for evaluating the impact of in-band crosstalk in optical DQPSK systems is developed and its results are confirmed by simulation. The crosstalk tolerance is reduced by 5dB in comparison to DPSK systems.