Recently Transformer-based pre-trained models have become a new paradigm for text summarization, and one of the milestones is PEGASUS. It suggests the pre-training objective Gap Sentences Generation designed for summarization. However, the ROUGE1-F1-based strategy in Gap Sentences Generation is unfavorable to Chinese text summarization, considering that unigram is not the basic semantic unit of Chinese in most cases. Furthermore, ROUGE1-F1 is based upon the co-occurrences of the unigrams other than distributional semantics such as word or sentence representations. Accordingly, the gap sentences are evaluated on a biased basis and thus the performance of GSG objective pre-training is hampered. In this work, we propose an improved GSG objective that upgrades the ROUGE1-F1-based strategy with TextRank. In TextRank, the texts are first segmented to extract semantically meaningful words and form distributional representations of sentences. The significance of a sentence is assessed in light of its cosine similarity with representations of the neighboring sentences in a graph. GSG with TextRank exhibits remarkable performance gains on the LCSTS dataset, and employment of R-Drop further boosts the improvements.