The main content of Text-guided Image Manipulation (TGIM) research is the use of textual information to modify the corresponding content in the input image. Based on generative adversarial networks (GAN), this research has achieved impressive manipulation performance. Nevertheless, the quality of image manipulation still needs to be further improved. In this paper, an attention-driven TGIM method is proposed to further improve the quality of image manipulation. Specifically, the proposed method uses an attention mechanism to fine-tune the whole process of image manipulation at the word level. Through attentional fine-tuning, the quality of image manipulation can be continuously improved to realize high-quality image manipulation effects. The proposed method is experimentally validated on a public Caltech-UCSD birds-200-2011 (CUB) dataset, and the qualitative and quantitative comparison results demonstrate the superior performance of the proposed method on TGIM. Compared to the existing TGIM methods, the proposed method improves the Inception Score (IS) by 22.6% and reduces Fréchet Inception Distance (FID) by 13.4%.