Temporal link prediction (TLP) is among the most important graph learning tasks, capable of predicting dynamic, time-varying links within networks. The key problem of TLP is how to explore potential link-evolving tendency from the increasing number of links over time. There exist three major challenges toward solving this problem: temporal nonlinear sparsity, weak serial correlation, and discontinuous structural dynamics. In this article, we propose a novel transfer learning model, called DNformer, to predict temporal link sequence in dynamic networks. The structural dynamic evolution is sequenced into consecutive links one by one over time to inhibit temporal nonlinear sparsity. The self-attention of the model is used to capture the serial correlation between the input and output link sequences. Moreover, our structural encoding is designed to obtain changing structures from the consecutive links and to learn the mapping between link sequences. This structural encoding consists of two parts: the node clustering encoding of each link and the link similarity encoding between links. These encodings enable the model to perceive the importance and correlation of links. Furthermore, we introduce a measurement of structural similarity in the loss function for the structural differences of link sequences. The experimental results demonstrate that our model outperforms other state-of-the-art TLP methods such as Transformer, TGAT, and EvolveGCN. It achieves the three highest AUC and four highest precision scores in five different representative dynamic networks problems.